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CHAPTER 1 



Introduction 



A computer is a machine designed to carry out operations automatically. Moreover, a 
computer is programmable, in the sense that it is easy to alter the particular task of a 
computer. Almost all computers are digital, in the sense that they manipulate discrete 
values, instead of continuous (analogue) ranges. Digital computers evolved significantly 
in the approximately 70 years since their appearance, and are now an essential part of 
most people's lives. 

At its core, a digital computer can be seen as a relatively simple machine that reads 
and stores values in memory, while performing arithmetic calculations on these values. 
The set of instructions to perform is itself also stored in and read from memory. Coupled 
with suitable input and output devices (such as a keyboard and a monitor), this allows 
a programmer to define a behaviour for the computer to execute. Modern computers can 
represent data in memory in different formats (such as integers, fractional numbers, or 
vectors, for instance), and perform various calculations (such as addition, trigonometry 
functions, and vector multiplication, for instance). Additionally, they can compare values 
and continue execution depending on the comparison ("if a certain memory value is 
zero then do this else do that", for instance). A program is then a seguence of these 
instructions. 

As advanced as the fundamental operations of modern digital computers might be, 
they are very far from what most users expect from a computer. Being able to add and 
subtract numbers is nice, but how do we use this to browse the internet, make a phone 
call, or to interpret a magnetic resonance exam? Even when computers can perform 
thousands of millions of primitive operations per second, we certainly do not want to 
express all tasks in terms of only simple operations. If we want to display some text 
on the screen, we want to say which characters should show up where; we do not want 
to have to write specially crafted numbers to a specific location in memory, which is 
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1 Introduction 

then interpreted by the screen as a sequence of intensity values for each of the pixels, 
eventually making up a sequence of characters! Naturally, we need to abstract from 
such details, and be able to formulate programs in a high-level fashion, specifying what 
should be done, and having the system decide how it should be done. 

Abstraction is ubiquitous in computer programming. To allow programmers to abstract 
from details, several programming languages have been developed. Such languages are 
a precise way to express commands to a computer, and eventually are translated into 
the only primitive operations that the computer understands. The automatic translation 
process from one programming language to another is the task of a compiler, itself a 
program which reads programs in one input language and outputs programs in another 
language. Compilers can be chained together, to collectively bring a program from a 
very high-level programming language (abstract and convenient for human reasoning) 
down to machine language (low-level and ready to be executed by the computer). 

The work of this thesis concerns a special class of very high-level programming lan- 
guages, namely statically-typed purely functional languages. In such languages, the 
computation to be performed is described by functions that take input and produce out- 
put. The evaluation of a program thus consists of evaluating calls to these functions. 
This paradigm is rather distant from the machine language at the core of the computer. 
The machine language deals with sequences of instructions, conditional operations, and 
loops. A functional language deals with function application, composition, and recur- 
sion. We choose functional programming as the starting point for our research because 
we believe it lends itself perfectly to express abstraction, leading to shorter and more 
understandable computer programs. This allows the programmer to express complex 
behaviour in a simple fashion, resulting in programs that are easier to adapt, compose, 
maintain, and reason about, all desirable properties for computer programs. 

We focus on one specific form of abstraction. Computer programs manipulate data, 
which can either be primitive machine data (such as integer or fractional numbers) or 
programmer-defined data (such as lists, trees, matrices, images, etc.). There is only a 
small number of primitive datatypes, but a potentially infinite number of programmer- 
defined data. The structure of the latter data depends on the problem at hand, and 
while some structures appear very often (such as sequences of values), others are truly 
specific to a particular problem. 

Some kind of functionality is generally desired for all types of data. Reading and 
storing files to the disk, for instance, is as important for machine integers as it is for 
complex healthcare databases, or genealogy trees. And not just reading and writing 
files: testing for equality, sorting, traversing, computing the length, all are examples of 
functionality that is often desired for all kinds of data. Most programming languages 
allow defining complex datatypes as a form of abstraction, but few provide good support 
for defining behaviour that is generic over data. As such, programmers are forced to 
specify this behaviour over and over again, once for each new type of data, and also to 
adapt this code whenever the structure of their data changes. This is a tedious task, 
and can quickly become time-consuming, leading some programmers to write programs 
to generate this type of functionality automatically from the structure of data. 

We think that a programming language should allow programmers to define generic 
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programs, which specify behaviour that is generic over the type of data. Moreover, it 
should automatically provide generic behaviour for new data, eliminating the need for 
repeated writing and rewriting of trivial code that just specialises general behaviour to a 
particular type of data. It should do so in a convenient way for the programmer, leading 
to more abstract and concise programs, while remaining clear and efficient. This leads 
us to the two research guestions we set out to answer: 

1 . There are many different approaches to generic programming, varying in complexity 
and expressiveness. How can we better understand each of the approaches, and 
the way they relate to each other? 

2. Poor runtime efficiency, insufficient datatype support, and lack of proper language 
integration are often pointed out as deficiencies in generic programming imple- 
mentations. How can we best address these concerns? 

We answer the first guestion in the first part of this thesis. We start by picking a number 
of generic programming approaches and define a concise model for each of them. We 
then use this model to formally express how to embed the structural representation 
of data of one approach into another, allowing us to better understand the relation 
between different approaches. The second part of this thesis deals with answering the 
second guestion, devoting one chapter to analysing and mitigating each of the practical 
concerns. 



1.1 Choice of programming language 



While functional languages are not the most popular for program development, they are 
ahead of popular languages in terms of the abstraction mechanisms they provide. We 
have also observed that popular languages, when adding new features, often borrow 
features first introduced and experimented with in functional languages. While we do 
not believe that the developments in this thesis will directly lead to widespread adoption 
of generic programming, we hope that they serve to further advance the field and provide 
further evidence that abstraction over the shape of data is an essential aspect of modern 
software development. 



All the code in this thesis is either in Haskell Peyton Jones 2003 , a statically- 
typed purely functional language, or Agda Norell 2007, a dependently-typed purely 
functional language. We use Agda in the first part of this thesis, since the first part deals 
with modelling typed representations, and many concepts that arise at the type level are 
easy to express in Agda because it is a dependently-typed language. Additionally, due 
to Agda's foundation in constructive type theory and the Curry-Howard isomorphism, we 
can use it for expressing formal proofs for some of our code. 

We use Haskell as the implementation language for the practical aspects of this thesis, 
in the dialect implemented by the Glasgow Haskell Compiler (GHC)pl Haskell is a fully 



1 http : //www . haskell . org/ghc/ 
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1 Introduction 

developed language with serious industrial applications, offering many advanced type- 
level programming features reguired for generic programming that are absent in most 
other languages. Haskell has a vibrant community of both users and researchers, which 
provides continuous feedback for what is relevant, necessary, and desired from generic 
programming in the wild. 

This thesis is written in literate programming style using lhs2TeX^]The resulting code 
files and other additional material can be found online at http : //dreixel . net /thesis 



1 .2 Relation with previously published material 



Some parts of this thesis are based on a number of previously refereed and published 
research articles. The author of this thesis is a main co-author in all of those articles. 
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Chapter 9 



on a single approach and to avoid using compilation flags that are pervasive through 
the whole program. Chapter 10 is a revised version of Magalhaes and Jeuring |201 1 



where a section originally discussing the use of unsafe casts has been replaced by a 
discussion on how to support datatypes with indices of custom kinds. Chapter TTj is an 
updated version of Magalhaes et a I. |2010a , reflecting the current support for generic 
programming in two Haskell compilers. 



1.3 Roadmap 



This thesis is divided in two parts, each part answering one research guestion. 

We start the first part with a gentle introduction to generic programming Chapter"2) , 
also introducing Agda along the way. We then present five generic programming ap- 
proaches: regular |Chapter~3) , pol yp {Chapter 4) , multirec {Chapter~5) , indexed 
{Chapter~6) , and instant-generics {Chapter 7| . We present each approach by first 
showing how it can be modelled in Agda, and then give its Haskell encoding. The de- 
scription of indexed lacks a Haskell encoding because Haskell is not (yet) capable of 
faithfully encoding this approach. On the other hand, we give more examples of generic 
functionality for indexed, namely a zipper for efficient navigation of data structures, 
and an algorithm for decidable eguality. Finally, [Chapter 8 presents a formal relation 
between the five approaches, showing, for instance, that the functionality of indexed 
can be used in other approaches. 

In the second part we focus on three main issues that often prevent a wider adoption of 
generic programming. Chapter 9] explores the potential for optimisation of generic pro- 
grams, with the aim of removing all runtime overhead of genericity through optimisation 



at compile time. Chapter 10 deals with extending datatype support for a representa- 
tive generic programming library, adding the ability to work generically with indexed 



^http : //www ■ andres- loeh ■ de/lhs2tex/ 
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1.3 Roadmap 

datatypes. Finally, |Chapter 1 1 1 presents a way to embed generic programming more 
seamlessly within Haskell, simplifying generic function definition and usage from the 
user's point of view. We conclude In |ChapterT2 with a few remarks on the future of 
generic programming. 
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Part I 

Generic programming libraries 
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CHAPTER 2 



Introduction to generic programming 



In this chapter we introduce the concept of generic programming through a series of 
examples in Agda, a dependency typed functional programming language. 

Colours and code highlighting 

Throughout this thesis we use this font (Helvetica) for code blocks. Keywords are high- 
lighted in bold, and we use three different colours in code blocks to distinguish identi- 
fiers. 

Haskell code 

In Haskell, there is a clear distinction between values, types, and kinds. Values are 
built with constructors, such as True, which we colour in blue; we do not colour numerals, 
characters, or strings: Just 3, (Right 'p' , Left "abc"). 

Datatypes, type synonyms, type variables, indexed families, and type classes are 
coloured in orange: Bool, Show a => Show [ a], type family PF a. 

Kinds and kind variables are coloured in green: *, * — > *, Constraint. For simplicity 
we do not colour the arrow operator, as it can appear at different levels. 

Agda code 

The distinction between values and types is less clear in Agda. Our convention is to use 
blue for constructors (e.g. refl), orange for identifiers of type Set (e.g. _=_), and green 
for identifiers of type Seti or (for simplicity) higher (e.g. Set). 
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2 Introduction to generic programming 



2.1 What is generic programming? 



The term "generic programming" has been used for several different but related concepts. 
Gibbons |2007 provides a detailed account which we summarise here. In general, all 
forms of generic programming deal with increasing program abstraction by generalising, 
or parametrising, over a concept. 



Value genericity Arguably the simplest form of generalisation is parametrisation by 
value. Consider first the type of natural numbers defined by induction: 

data N : Set where 

ze : N 

su : N -y N 

This declaration defines a new type N that is inhabited by the ze element, and an 
operation su that transforms one natural number into another. We call ze and su the 
constructors of the type N, as they are the only way we can build inhabitants of the 
type. 

Consider now an operator that computes the sum of two natural numbers: 

_+_ : N — > N — > N 

m + ze = m 

m + (su n) = su (m + n) 

The _+_ operator (with underscores denoting the position of its expected arguments) is 
our first example of generalisation, as it defines an operation that works for any two 
natural numbers. Value abstraction is generally not called "generic programming", as 
it is nearly universally present in programming languages, and as such is not consid- 
ered special in any way. It serves however as a very basic example of the concept of 
abstraction: one that is crucial in programming languages in general, and in generic 
programming in particular. 



Type genericity Suppose now we are interested in handling seguences of natural 
numbers. We can define such a type inductively in Agda as follows: 

data NList : Set where 
nil : NList 

cons : N -> NList -> NList 

Now we can define lists of natural numbers. For example, nil encodes the empty list, and 
cons 0 (cons 1 (cons 2 nil)) encodes the list of natural numbers from zero to two. Note 
that for convenience we are using numerals such as 0 and 1 instead of ze and su ze. 

With NList we are limited to encoding seguences of natural numbers, but the concept 
of a list is not in any way specific to the naturals. Since Agda has a polymorphic type 
system, we can also define a "type generic" list datatype: 



14 



2.1 What Ls generic programming? 



data[_](cr : Set) : Set where 

0 :[«] 
:: : a — > \ a ] — > f a ] 



This parametric [_] type encodes lists of any Set a; in particular, [N] is eguivalent (in 
the formal sense of Section 2.2.3) to the NList type we have just defined. We use the 



symbol [] to represent the empty list, while is the operator that adds an element to 
a list. 

This form of parametrisation is often called simply "generics" in languages such as 
Java Bracha et a I. 2001 and C# |Kennedy and Syme 2001 , but it is more commonly 



referred to as "parametric polymorphism" in Haskell or Agda. 

Functions operating on polymorphic datatypes can also be polymorphic. For instance, 
a function that computes the length of a list does not care about the particular type of 
elements in the list: 

length : {a : Set} — > [ a] — > N 

length [] =0 

length (h :: t) = 1 + length t 

Functions such as length are called polymorphic functions. In Agda, type arguments to 
functions have to be explicitly introduced in the type declarations. So we see that length 
takes a Set a, a list of as, and returns a natural number. The curly braces around a 
indicate that it is an implicit argument, and as such it does not appear as an argument 
to length. The Agda compiler tries to infer the right implicit arguments from the context; 
when it fails to do so, the programmer can supply the implicit arguments directly by 
wrapping them in curly braces. 



Gibbons distinguishes also "function genericity", where the polymorphic arguments of 
functions can themselves be functions. A simple example is the function that applies a 
transformation to all elements of a list: 

map : {a $ : Set} -»■ (a jS) [a] ->• [0] 

mapf[] = 0 

map f (h :: t) = f h :: map f t 

This function transforms a list containing as into a list of the same length containing /3s, 
by applying the parameter function f to each of the elements. The map function is akin 
to the loop operation in imperative programming languages, and can be used for many 
distinct operations. For example, to increment each of the naturals in a list we can use 
the map su function, and if we have a function toChar : N — > Char that converts natural 
numbers to characters, we can transform a list of naturals into a string by applying the 
function map toChar. These are simple examples, but they already reveal the power 
of abstraction and generalisation in reducing code duplication while increasing code 
clarity. 

Structure genericity When we pair polymorphism with abstract specifications we get 
to the so-called "C++ generics". The Standard Template Library |Austern 1999 is the 
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primary example of structure genericity in C++, which consists of specifying abstract 
data structures together with standard traversal patterns over these structures, such as 
iterators. The abstract structures are containers such as lists, vectors, sets, and maps, 
whereas the iterators provide support for accessing the contained elements in a uniform 
way. Unlike our [_] datatype, these abstract containers do not specify constructors; the 
only way to operate on them is through the iterators. 

This form of structure genericity is commonly called "generic programming" within 
the C++ community Siek et a I. 2002, and is also known as "overloading" in other 
languages. In Haskell, a similar programming style can be achieved through the use of 
type classes, and in Agda through the use of module parameters. This is, however, not 
the style of generic programming that is the subject of this thesis. Although structure 
genericity allows for easy switching between different types of containers, it does not 
necessarily reduce code duplication, for instance, as similar containers might still have 
to separately implement similar operations, without any possibility of code reuse. 



Stage genericity A metaprogram is a program that writes or manipulates other pro- 
grams. In combination with some form of staging or splicing, this allows for programs 
that manipulate themselves, or perform computations during type checking. Template 
Haskell Sheard and Peyton Jones 2002 is an implementation of metaprogramming in 
GHC; Agda has a reflection mechanism with similar expressiveness]^] 

Metaprogramming mechanisms can be very expressive and allow for defining generic 
programs in all the senses of this section. For instance, one can define a Template 
Haskell program that takes a container type and then defines an appropriate length 
function for that type. This could in principle be done even in a non-polymorphic lan- 
guage. However, encoding generic behavior through metaprogramming is a tedious and 
error-prone task, since it involves direct manipulation of large abstract syntax trees. 



Shape genericity The type of genericity we focus on arises from abstracting over the 
shape of datatypes. Consider a datatype encoding binary leaf trees: 

data Tree (a : Set) : Set where 
leaf : a — » Tree a 
bin : Tree a — > Tree a — > Tree a 

For the list datatype we have seen length and map functions. However, these functions 
are not specifically exclusive to lists; we can egually well define them for trees: 

length : {a : Set} -> (Tree a) -> N 

length (leaf x) = 1 

length (bin I r) = length I + length r 

map : {a : Set} -»• (a -> j8) -> (Tree a) -> (Tree 0) 

map f (leaf x) = leaf (f x) 

map f (bin I r) = bin (map f I) (map f r) 

1 http: //wiki .portal . Chalmers . se/agda/pmwiki .php?n=Ref erenceManual . Reflection 
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In fact, length and map are only two of many functions that are generic over the structure 
of datatypes. Note how the structure of the functions Is determined by the datatype 
Itself: for length, we return 0 for constructors without arguments, 1 at any occurrence 
of the parameter, and call length recursively at the recursive positions of the datatype 
In question, summing the results obtained for all the components of each constructor. 
For map, we keep the structure of the datatype, applying the mapping function to the 
parameters and recursively calling map at recursive positions. 

This Is the type of generic programming we are Interested In: programs that generalise 
over the shape of datatypes. This style of programming Is also called "datatype-generic", 



"polytypic", and "type-Indexed" programming Dos Rels and Jarvl 2005. From this point 



on, when we refer to "generic programming" we mean this form of generlclty. 

So far we have only shown two definitions of both map and length, and argued that 
they really should be a single function each. We now proceed to show precisely how 
that can be done. 



2.2 Categorical background 

Generic programming arose from the categorical concepts of F-algebras and homomor- 
phlsms. A detailed Introduction to category theory and how It relates to functional 
programming In general and datatypes In particular Is outside the scope this thesis; the 
reader Is referre d to |Backhouse et al.|[l999] , |Blrd and Moor|pr997] , |Fokkinga]|1 992] , and 
Meertens 1995 for a precise account of these foundations. Here we focus on a practical 
encoding of functors for constructing datatypes, up to simple recursive morphlsms and 
some of their laws. 



2.2.1 Polynomial functors 

The key aspect of generic programming Is In realising that all datatypes can be expressed 
In terms of a small number of primitive type operations. In order to handle recursion, as 
we will see later, the primitive types are all functors: 

Functor : Seti 
Functor = Set -» Set 

For the purposes of this section, we define a Functor to be a function on sets^j The 
Functor type Is not a Set since It contains sets Itself, therefore It must be In Set 
Chllpala |201 2 for more Information on hierarchical universes 



see 

Consider now the fol- 



lowing encoding of a lifted sum type: 



data (4> <A 

inji : <p p - 
inj 2 : ip p - 



: Functor) (p 

(0 e <//) p 
(</> © <P) p 



Set) : Set where 



2 Categorlcally speaking, this is the functor operation on objects. On the other hand, the Haskell Functor 
class represents the categorical functor operation on morphisms. Normallg functors also have associated 
laws, which we elide for the purposes of this section, but revisit in [Section 6.2| 
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This is a "lifted" type because it deals with Functors, instead of just sets. Its arguments 
<p and ip are also Functors, and the sum <p © tp is itself of type Functor. It is called a 
"sum" because it shares many properties with the usual algebraic sum; another common 
name is "disjoint union". Its dual is the lifted product: 

data _®_ ((f) ip : Functor) (p : Set) : Set where 

_,_ : <pp^ipp^(<p®ip)p 

Again, the name "product" refers to its similarity to the algebraic product, or the set 
product. Finally, consider the encoding of a lifted unit type: 

data 1 (p : Set) : Set where 
1 : 1 p 

We call this a "unit" type because it has a single inhabitant, namely the 1 constructor. 
It is also lifted, as evidenced by its argument p. 

These basic three datatypes, which we call "representation" functors, can be used to 
encode many others. Consider a very simple datatype for encoding Boolean values: 

data Bool : Set where 
true : Bool 
false : Bool 

To encode a Boolean we have a choice between two possible values: true and false. If 
we ignore the constructor names, this is eguivalent to the sum of two units: 

Bool F : Functor 
Bool F = 1 © 1 

The type BooIf p also has only two inhabitants, namely inji 1 and inj2 1- The alge- 
braic nature of our representation functors also becomes evident: the type 1 has one 
inhabitant, and the type 1 © 1 has two inhabitants. 



2.2.2 Recursion 

The N datatype of Section 2.1 is a choice between ze or su. We can also encode this 



datatype as a sum, but we are left with the problem of dealing with recursion. Basically, 
we want to encode the following recursive eguation: 

Nat F = 1 © Nat F 

For this we first need an identity functor: 

data Id (p : Set) : Set where 
id : p —>■ Id p 

This datatype, which simply stores the parameter p, will be used in place of the recursive 
occurrences of the datatype being coded. For N we thus get: 
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2.2 Categorical background 



Nat F : Functor 
Nat F = 1 © Id 

The functorLal nature of our representation types Ls now clear. The type NatF Is not 
recursive; however, with the right Instantiation of the parameter p, we can define N In 
terms of Natp. For this we need a fixed-point operator: 

data fj (</> : Functor) : Set where 

(_) : <p [f, ft -» n <S> 

out : {(/> : Functor} -> /v cf> — > </> (/J </>) 
out ( x } = x 

This primitive recursive operator, with constructor (_) and destructor out, can be used 
to tie the recursive knot of our representation type Natp. In fact, fj NatF Is egulvalent to 
N, as It encodes the Infinite expansion 1 © 1 © 1 © . . ., which we obtain by replacing 
each occurrence of Id In NatF with the expansion of Natp Itself. 

The type [_] of lists can be encoded similarly, but first we need one more represen- 
tation type for constants: 

data K (a p : Set) : Set where 
k : a — > K a p 

The type K Is a lifted constant operator, which we use for encoding the type parameter 
of lists. The encoding of lists [_] F uses all the representation types we have seen: 

[_]f : Set -> Functor 
[a] F = 1 © (K a ® Id) 

The [] constructor Is encoded as a unit, hence the 1, whereas the :: constructor has two 
arguments, so we use a product. The first argument Is a parameter of type a, which we 
encode as a constant, and the second argument Is again a list, so we use Id. The fixed 
point jj [a]f corresponds to the algebraic expansion 1 © (K a <8> (1 © (K a ® •••)))' 
which Is egulvalent to [a]. 

2.2.3 Isomorphisms 

We have mentioned that some types are equivalent to others without formally defining 
this egulvalence. When we say two types are egulvalent we mean that they are iso- 
morphic. The concept of Isomorphism can be succinctly summarised as an Agda record 
type: 

record ~ (cr : Set) : Set where 
field 

from : a — > j3 
to : /? — > a 
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2 Introduction to generic programming 

isoi : V x — > to (from x) = x 
iso2 : V x — » from (to x) = x 

An isomorphism a ~ (3 is a tuple of four components: two functions that convert between 
a and /?, and two proofs isoi and is02 that certify the cancellation of the composition of the 
conversion functions. We call iso-j and is02 proofs because their type is a propositional 
eguality; the _=_ type is defined as follows: 

data = {a : Set} (x : a) : a — > Set where 
refl : x = x 

The only way to build a proof of propositional eguality refl is when the arguments to 
_=_ are egual. In this way, if we can give a definition for isoi, for instance, we have 
proven that to (from x) eguals x. 

Previously we have stated that N and ^/ Natp are eguivalent. We can now show that 
they are indeed isomorphic. We start by defining the conversion functions: 

fromN : N -> \i Nat F 

fromN ze = ( inji 1 ) 

fromN (su n) = ( inj 2 (id (fromN n)) ) 

toN : \i Nat F ->■ N 

toN ( inj, 1 } = ze 

toN ( inj 2 (id n) } = su (toN n) 

The conversion from N starts by applying the fixed-point constructor (_), then injecting 
each constructor in the respective component of the sum, and then stops with a 1 for 
the ze case, or proceeds recursively through id for the su case. The conversion to N is 
trivially symmetrical. 

We are left with proving that these conversions are correct. For this we have to build 
a proof, or an element of the _=_ type. We already know one such element, refl, which 
can be used when Agda can directly determine the structural eguality of the two terms. 
However, most often the arguments are not immediately structurally eguivalent; then we 
need to convince the compiler of their eguality by building an explicit proof derivation. 
For this we can use combinators like the following: 

sym : {a f3 : Set} — > a = f3 — > /3 = a 
sym refl = refl 

trans : {cr/3)/:Set}— >cr = /?— >/?=y— >a=y 
trans refl refl = refl 

cong : {a/?:Set}{xy: a } — > (f : a— >/?)— >x = y— >fx = fy 
cong f refl = refl 

The first two combinators, sym and trans, encode the usual relational concepts of sym- 
metry and transitivity. The cong combinator (short for congruence) is generally used 
to move the focus of the proof over a Functor. We use it in the isomorphism proof for 
naturals: 
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2.2 Categorical background 



isoNi : (n : N) -> toN (fromN n) = n 

isoNi ze = refl 

isoNi (su n) = cong su (isoNi n) 

isoN 2 : (n : p Nat F ) -* fromN (toN n) = n 
isoN 2 ( inji 1 } = refl 

isoN 2 ( inj 2 (id n) } = cong (A x — > ( inj 2 (id x) }) (isoN 2 n) 

For the ze case we can directly use refl, but for the su case we have to proceed recursively, 
using cong to move the focus of the proof over the functor to the recursive position. 
We now have all the components necessary to form an isomorphism between N and 

p Nat F : 

isoN : N ~ p Nat F 

isoN = record { from = fromN; to = toN 

; isoi = isoN-i ; iso 2 = isoN 2 } 

The isomorphism between [a] and /J [a]f can be constructed in a similar way. 



2.2.4 Recursive morphisms 



Many generic functions are instances of a recursive morphism Meijer et a I. 1991 . These 
are standard ways of performing a recursive computation over a datatype. We start with 
catamorphisms, based on the theory of initial F-algebras. Categorically speaking, the 
existence of an initial F-algebra out -1 : F a — > a means that for any other F-algebra 
0 : F (3 —* there is a unigue homomorphism from out -1 to <fi. This information can 
be summarised in a commutative diagram: 



F a 



l 3< i F ^ 

If we invert the direction of out -1 and instantiate it to the type ii Nat F — > Nat F (/v Nat F ) 
we get the definition of catamorphism for natural numbers: 

li Nat F — ► Nat F (p Nat F ) 

/S < Nat F j8 



<\ t) 
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To be able to encode this diagram as a function, we are still lacking a functorial 
mapping function, used in the arrow on the right of the diagram. For the natural numbers, 
we are looking for a function with the following type: 

mapN : {a & : Set} -> (a ->• /S) -> Nat F a -> Nat F p 

This function witnesses the fact that Nat F is a functor in the categorical sense, mapping 
a morphism between Sets to a morphism between NatpS. It is also a generic function, in 
the sense that its definition is determined by the structure of the datatype it applies to. 
Since Natp is 1 © Id, to define mapN we first need to define map on units, sums, and 
identity: 

map t : {p pi : Set} — > (p — > pi) -* 1 p — > 1 pi 
mapn _ 1 = 1 

map ffi :{(/><//: Functor} {p pi : Set} — > 

[<t> P -» </> P') -»■ (<A P «/* P') -» (</> © P -» (</> © P' 
map ffi f g (inji x) = inj-i (f x) 
map ffi f g (inj 2 x) = inj 2 (g x) 

map| d : {p pi : Set} — > (p —> pi) —> Id p — > Id p/ 
map| d f (id x) = id (f x) 

Units do not contain any p, so we do not map anything. We choose to define mapj 
taking the function argument for consistency only; it is otherwise unnecessary. The fact 
that we do not use it in the definition is also made clear by pattern-matching on an 
underscore. For sums we need to know how to map on the left and on the right. For an 
identity we simply apply the function. 
We can now define mapN generically: 

mapN : {a & : Set} -> (a -> /S) -> Nat F a -> Nat F p 
mapN f = map ffi (mapj f) (mapi d f) 

In the same style as for Nat F , we also have a mapping function for [ a ] F , or any other 
functor written using the polynomial functors we have introduced so far, because these 
(aptly named) polynomial functors are functorial. The definition of catamorphism for N 
follows: 

cataN : : Set} -> (Nat F jS 0) \i Nat F -» 0 
cataN </> = </> o mapN (cataN <p) o out 

Note how the definition directly follows the diagram, using function composition (repre- 
sented in Agda by the operator _o_) for composing the three arrows on the rightrl 

3 The definition of cataN does not pass Agda's termination checker. For practical purposes we disable the 
termination checker and do not guarantee termination for our functions. |Section 8.4| presents a small 
discussion on the implications of this decision on our proofs. 
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2.2 Categorical background 



In this chapter we have defined the term "generic programming" and seen some con- 
crete examples. However, we have manually defined structural representation types, 
and manually instantiated generic functions. In the next chapter we show a practical 
encoding of these same concepts in a real library, including the necessary infrastructure 
to derive generic definitions of map and catamorphism automatically. 
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CHAPTER 3 



The regular library 



In this chapter we introduce the regular library for generic programming. We start by 
defining a concise model of the library's core in Agda, and then show practical aspects of 
its implementation in Haskell. This is the general structure of each of the five chapters 
introducing generic programming libraries in this thesis: we first introduce an Agda 
model of the library, and then show the details of the Haskell implementation ^| The 



Agda model focuses on core concepts such as the generic view Holdermans et a I. 



2006 



used and the datatype support offered. We show it first as it serves also as a theoretical 
overview of the library at hand. The Haskell encoding deals with more practical aspects 
such as ease of use and performance. 



3.1 Agda model 



regular is a simple generic programming library, originally developed to support a 
generic rewriting system |Van Noort et al. 2008 . It has a fixed-point view on data: the 
generic representation is a pattern-functor, and a fixed-point operator ties the recursion 
explicitly. In the original formulation, this is used to ensure that rewriting meta-variables 
can only occur at recursive positions of the datatype. This fixed-point view on data is 
very close to the categorical view on datatypes introduced in the previous chapter. The 
library name derives from the concept of a regular datatype, which is a type that can 
be represented as a potentially recursive polynomial functor. This definition excludes 
exponentials (functions), or nested datatypes Bird and Meertens 1998, amongst others, 
which regular indeed does not support. 



1 Chapter 6 follows a slightly different structure; the reason for the different treatment is explained there. 
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3 The regular library 

We model each library by defining a Code type that represents the generic universe, 
and an interpretation function [_] that maps codes to Agda types. The universe and 
interpretation for regular follow, side by side: 



data Code : Set where 
U : Code 
Code 

(FG : Code) 
i_ : (F G : Code) 



Code 
Code 



[J : Code (Set Set) 

[U J a = T 

I I ] a = a 

[ F © G ] a = [ F ] a i+l [G] a 

JF® G] a = JF] a x [Gj a 



We have codes for units, identity (used to represent the recursive positions), sums, and 
products. The interpretation of unit, sum, and product relies on the Agda types for unit 
(T), disjoint sum (_W_), and non-dependent product (_x_), respectively. The interpre- 
tation is parametrised over a Set a that is returned in the I case. We omit a case for 
constants for simplicity of presentation only. 

As before, we need a fixed-point operator to tie the recursive knot. We define a // 
operator specialised to regular, taking a Code as argument and computing the inter- 
pretation of its least fixed point: 



data fj (F : Code) 
(J : !F](/iF) 



Set where 



In Section 2.2.4 we have introduced the recursive morphisms, and how they can be 
defined from a map function. In regular, this function lifts a mapping between sets 
a and ft to a mapping between interpretations parametrised over a and /?, simply by 
applying the function in the I case: 



map : (F : Code) —. 

map U f _ 
map I f x 

map (F © G) f (inj! x) 
map (F © G) f (inj 2 y) 
map (F ® G) f (x , y) 



= tt 



Set} -> (a $) 



F] a 



F]0 



f x 

in^ (map F f x) 
inj 2 (map G f y) 
map F f x , map G f y 



Note that tt is the only constructor of the T type, inji and inj 2 are the constructors of 
_W_, and _,_ is the constructor of x . 

We can now give a truly generic definition of catamorphism: 



cata : {a : Set} -> (F : Code) -> ([ F J a 
cata F f ( x ) = f (map F (cata F f) x) 



This definition of cata works for any regular Code, using the generic definition of map. 

Datatypes can be encoded by giving their code, such as NatC for the natural numbers, 
and then taking the fixed point: 
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3.2 Haskell Implementation 

NatC : Code 
NatC = U 0 I 

Hence, a natural number Is a value of type \i NatC; In the example below, aNat encodes 
the number 2: 

aNat : \i NatC 

aNat = ( inj 2 ( inj 2 ( inji tt } } } 



3.2 Haskell implementation 

In Haskell, the universe and Interpretation of regular is expressed by separate datatypes: 

data U a = U 

data I a = I a 

dataK£ a = K jS 

data (<p :+: (//) a = L (<p a) | R (ijj a) 

data (c/> :x: ip) a = (pa :x: ip a 

These definitions are not too different from the code and interpretation in the Agda 
model. The left-hand side of the datatype declarations are the codes, whereas the 
right-hand side is the interpretation. We add a K code for embedding arbitrary constant 
types in the universe; this is essential in a practical implementation to handle primitive 
types (such as Int or Char). 

Note that while we use only these representation types to represent regular codes, 
nothing prevents us from writing types such as (Maybe :+: U) Int, for instance, even though 
this does not make sense since Maybe is not a representation type. In the Agda model 
these mistakes are impossible because Maybe is not of type Code and attempting to use 
it as such would trigger a type error. Note also that since the original implementation 
of the regular library in Haskell there have been new developments that do allow 



preventing this type of errors Yorgey et al. 2012, but these have not yet been used to 
improve regular. 

The library also includes two representation types for encoding meta data, such as 
constructor names and fixities, that are necessary for defining generic functions such 
as show and read. For conciseness we do not make use of those in our description; 
in ISection 1 1 .2.21 we describe in detail how to handle meta-data information within 
representation types. 



3.2.1 Generic functions 

Generic functions are written by induction over the universe types. To give instances 
for the different types we use a type class. For instance, the map function for regular 
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3 The regular library 



corresponds directly to the standard Haskell Functor class^J 

class Functor cf> where 

fmap :: (a - — > /?) -* <p a —> cf> (3 

The implementation of the fmap function in regular is given by defining an instance for 
each of the representation types: 



instance Functor U where 
fmap _ U = U 

instance Functor I where 
fmap f (I r) = I (f r) 

instance Functor (K a) where 
fmap _ (K x) = Kx 

instance (Functor (j>, Functor <//) => Functor (cj> :+: tp) where 
fmap f (L x) = L (fmap f x) 
fmap f (R x) = R (fmap f x) 

instance (Functor </>, Functor tp) => Functor (0 :x: tp) where 
fmap f (x :x: y) = fmap f x :x: fmap f y 

The instances are very similar to the map function in our Agda model, albeit more verbose 
due to the need of writing the instance heads. 



3.2.2 Datatype representation 

To convert between datatypes and their generic representation we use another type 
class. Since regular encodes the regular types, we call this class Regular too: 



class Regular a where 



type PF a 

from 
to 



a 

PF a a 



PF a a 

a 



The class has an associated datatype |Schrijvers et al. 2008 PF, which stands for pattern 
functor, and is the representation of the type in terms of our universe codes, regular 
adopts a shallow encoding, so we have a one-layer generic structure containing user 
datatypes at the leaves. This is clear in the type of from, for instance: it transforms a 
user datatype a into a pattern functor (PF a) a, which is the generic representation of 
a containing a types in the I positions. 
As an example, consider the definition and generic representation of natural numbers: 



Functor instances In Haskell are often used to allow mapping a function to the elements contained within 
a type constructor. On the other hand, the Functor Instances for regular types map over the recursive 
positions of a datatype. 
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3.2 Haskell Implementation 



dataN = Ze I Su N 



instance Regular N where 
type PF N = U :+: I 

from Ze = LU 
from (Su n) = R (I n) 

to (L U) = Ze 
to (R (I n)) = Su n 

The pattern functor of N is a choice between unit (for Ze) and a recursive position (for 
Su). The conversion functions are mostly unsurprising, but note that in R (I n), the n 
is of type H, and not of type PF N N. Note also that the conversion functions are not 
recursive; to fully convert a value to its generic representation we need to map from 
recursively: 

data 4> — In (0 [fj 0)) 

deepFromN :: N -> \i (PF N) 

deepFromN = In o fmap deepFromN o from 

With a fixed-point operator we can express the type of a fully converted value, namely 
fj (PF N). In practice, however, regular does not make use of deep conversion functions. 



3.2.3 Generic length 



In Section 2.1 we promised to show generic definitions for functions map and length that 



would work on multiple types. We have already shown map, and we can show length 
too: 

class LengthAlg 4> where 
lengthAlg :: <p Int -> Int 



instance LengthAlg U where 
lengthAlg _ =0 

instance LengthAlg I where 
lengthAlg (In) = n + 1 

instance LengthAlg (K a) where 
lengthAlg _ =0 

instance (LengthAlg cf>, LengthAlg <//) => LengthAlg (0 :+: <//) where 
lengthAlg (L x) = lengthAlg x 
lengthAlg (R x) = lengthAlg x 

instance (LengthAlg </>, LengthAlg <//) => LengthAlg (</> :x: ip) where 
lengthAlg (x :x: y) = lengthAlg x + lengthAlg y 

The lengthAlg function operates on generic representations and computes their length. 
We assume we already know the length of each recursive position, and add up the length 
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3 The regular library 



of multiple arguments. However, lengthAlg is only the algebra of the computation. To 
apply it recursively over a term we use the catamorphism, this time in Haskell: 

cata :: (Regular a, Functor (PF a)) => (PF a f3 -> /?) -> a -> /? 
cata f x = f (fmap (cata f) (from x)) 

We now have all the elements in place to define generic length for every Regular 
datatype: 

length :: (Regular a, Functor (PF a), LengthAlg (PF a)) => a — > Int 
length = cata lengthAlg 

We have only seen one Regular instance, namely for N; as an example, length Su (Su (Su Ze)) 
evaluates to 3. Consider now a Regular instance for lists: 

instance Regular [ a } where 

type PF [a] = U :+: (K a :x: I) 

from [] = L U 

from (h : t) = R (K h :x: 1 1) 

to (L U) = [] 

to(R(Kh:x: It)) = h:t 

With this Regular instance for lists we can compute the length of lists, generically. For 
instance, length [()] evaluates to 1, and length [0,2,4] to 3. 



We have revisited the foundations of generic programming from Chapter 2 and cast 
them in the setting of a concrete, simple generic programming library, supporting only 
regular datatypes. In the following chapters we will present more complex approaches 
to generic programming, with increased flexibility and datatype support. 
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CHAPTER 4 



The polyp approach 



polyp Jansson and JeurLng 1997 Ls an early pre-processor approach to generic pro- 
gramming. As a pre-processor, It takes as Input a file containing Haskell code and some 
special annotations, and outputs a Haskell file, ready to be handled by a full Interpreter 
or compiler. Haskell evolved since 1997 and It Is now possible to encode polyp as a 
library In a similar fashion to regular. In fact, polyp has a very similar generic view to 
that of regular, only that It abstracts also over one datatype parameter, In addition to 
one recursive position, polyp therefore represents types as blfunctors, whereas regular 
uses plain functors. 



4.1 Agda model 

The encoding of polyp's universe In Agda follows: 



data Code : Set where [_] : Code ->• (Set Set Set) 



u 


Code 






[u 


] 


a/3 = 


T 








p 


Code 






[P 


] 


a/3 = 


a 








I 


Code 






II 


] 


a(3 = 












(FG : 


Code) 


Code 


[Ffl 




a/3 = 


in 


a $ W 


[G] 


a 13 




(FG : 


Code) 


Code 


[F(S 


sg; 


a/3 = 


in 


a /3 x 


[G] 


a 13 


_®_ 


(FG : 


Code) 


-> Code 


[f@g; 


a/3 = 


JJF 


[Gjcr 


0 





In the codes, the only differences from regular are the addition of a P code, for the 
parameter, and a code _©_ for composition. The Interpretation Is parametrised over 
two Sets, one for the parameter and the other for the recursive position. Composition Is 
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interpreted by taking the fixed-point of the left bifunctor, thereby closing its recursion, 
and replacing its parameters by the interpretation of the right bifunctor. The fixed-point 
operator for polyp follows: 

data \i (F : Code) (a : Set) : Set where 
(_) : [ F ] a (/i F a) -»■ fj F a 

There is at least one other plausible interpretation for composition, namely interpreting 
the left bifunctor with closed right bifunctors as parameter (J F ] (/j G a) /?), but we give 
the interpretation taken by the original implementation. 

This asymmetric treatment of the parameters to composition is worth a detailed dis- 
cussion]^] In polyp, the left functor F is first closed under recursion, and its parameter 
is set to be the interpretation of the right functor G. The parameter a is used in the 
interpretation of G, as is the recursive position f3. Care must be taken when using 
composition to keep in mind the way it is interpreted. For instance, if we have a code 
for binary trees with elements at the leaves TreeC, and a code for lists ListC, one might 
naively think that the code for trees with lists at the leaves is TreeC ® ListC, but that 
is not the case. Instead, the code we are after is (ListC © P) © (I <8> I). Although the 
resulting code guite resembles that of trees, this composition does not allow us to reuse 
the code for trees when defining trees with lists. The indexed approach {Chapter~6 
has a more convenient interpretation of composition; this subtle difference is revealed 
explicitly in our comparison between polyp and indexed {Section 8.2.3| . 

The fixed-point operator takes a bifunctor and produces a functor, by closing the 
recursive positions and leaving the parameter open. The map operation for bifunctors 
takes two argument functions, one to apply to parameters, and the other to apply to 
recursive positions: 

map : { a p y 5 : Set} 
(F : Code) -> (a 
map U f g tt = 
map P f g x = 
map I f g x = 

map (F © G) f g (inji x) = 
map (F © G) f g (inj 2 y) = 
map (F <g> G) f g (x , y) = 
map (F © G) f g ( x } 

To understand the case for composition recall its interpretation: the parameters are 
G bifunctors, which we handle with map G, and the recursive occurrences are again 
compositions, so we recursively invoke map (F © G). 

A map over the parameters, pmap, operating on fixed points of bifunctors, can be built 
from map trivially: 

1 ln fact, it is even questionable if polyp encodes true composition, or only a form of functor combination. 
We will, however, stick to the name "composition" for historical purposes. 



♦ J8) -(y - ^iFjoy - [ F ] j8 5 
tt 

f x 

gx 

in^ (map F f g x) 

inj 2 (map G f g y) 

map F f g x , map G f g y 

( map F (map G f g) (map (F © G) f g) x } 
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4.2 Haskell Implementation 

pmap : {a jS : Set} (F : Code) -4 (o ^ jS) ^ pFcr ^ (iFjS 
pmap F f ( x ) = ( map F f (pmap F f) x ) 

As an example encoding In polyp we show the type of non-empty rose trees. We first 
encode lists in ListC; rose trees are a parameter and a list containing more rose trees 
(RoseC): 

ListC : Code 

ListC = U © (P <g> I) 

RoseC : Code 

RoseC = P ® (ListC © I) 

The smallest possible rose tree is sRose, containing a single element and an empty list. 
A larger tree IRose contains a parameter and a list with two small rose trees: 

sRose : fj RoseC T 
sRose = ( tt, ( inj! tt } } 

IRose : \i RoseC T 

IRose = ( tt, ( inj 2 (sRose, ( inj 2 (sRose, ( inj 1 tt }))))) 



4.2 Haskell implementation 



An earlier encoding of polyp directly in Haskell used Template Haskell Norell and 



Jansson 2004. Nowadays we can encode polyp in a way similar to regular using 



type families. We first show the representation types: 

data U p a — U 
data I p a = I {unl :: a} 
data P p a = P p 
data K 0 p a = K /? 

data (4> :+: <//) p a — L ((/> p a) | R (ip p a) 
data (<fi :x: ijj) p a — <fi p a :x: ijj p a 
data (5 :o: cf>) p a = Comp (/y 5 (0 p a)) 

The differences from regular are: 

• We carry another type parameter around, p, which stands for the parameter that 
polyp abstracts over. Types with multiple parameters are supported, but only one 
parameter is not treated as a constant. So only this parameter p can be mapped 
over, for instance. Recall that regular treats all parameters as constants (with 
the K representation type). 

• The parameter p is used in the P representation type, and carried around other- 
wise. 
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4 The polyp approach 



• A new representation type :o: stands for code composition. Its constructor has 
the same asymmetric treatment of parameters as the Agda model we have shown 
previously. The fixed-point operator with one parameter in Haskell is: 

data \i 5 p = In {out :: 5 p [fj d p)} 

Note also that we define both I and [i as records. This is only to simplify later 
definitions by using the record selector function instead of pattern matching. 



4.2.1 Generic functions 

As before, generic functions are written by induction over the universe types using a 
type class. We show a bifunctorial map function as example: 

class Bimap <p where 

bimap :: (a — > /?) — > (y —y 5) — > <fi a y — > cf> 5 

instance Bimap U where 
bimap U = U 

instance Bimap P where 
bimap f _ (P x) = P (f x) 

instance Bimap I where 
bimap _g (I r) = I (g r) 

instance Bimap (K a) where 
bimap (K x) = K x 

instance (Bimap cp, Bimap <//) => Bimap (cf> :+: (//) where 
bimap f g (L x) = L (bimap f g x) 
bimap f g (R x) = R (bimap f g x) 

instance (Bimap cj), Bimap <//) => Bimap (cf> :x: ifj) where 
bimap f g (x:x:y) = bimap f g x :x: bimap f g y 

instance (Bimap 5, Bimap 4>) => Bimap (5 :o: 0) where 
bimap f g (Comp x) = Comp (pmap (bimap f g) x) 

The case for composition relies on a function to map over the parameters of a fixed point: 

pmap :: (Bimap 5) => (a — > /?) —> ^/ 5 a —> fj 5 $ 
pmap f = In o bimap f (pmap f) o out 

When specialised to lists, pmap implements the standard Haskell map. Other generic 
functions can be defined similarly, either as an algebra (like lengthAlg in Section 3.2.3) , 
or by handling recursion directly in the I instance. 



4.2.2 Datatype representation 

We again use a type class to convert between datatypes and their generic representation 
as polyp bifunctors: 
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class PolyP (5 :: * — » *) where 
type PF 5 :: * — > * — > * 

from :: 5 p — > /j (PF 5) p 
to :: a/ (PF 5) p —> 5 p 

In the original description the equivalent of this class was called Regular; we rename It to 
PolyP to avoid confusion with the regular library. There are two Important differences 
from Regular: 

• The class parameter 5 has kind * — > *. So we represent only the container types, 
such as lists. Types of kind * cannot be directly represented In this class; In 
Section 1 1 ,2| we show how to support both kinds of datatype with a single set of 
representation types. 

• To be more faithful to the original description, we opt for a deep encoding, unlike 
the shallow encoding of regular. So the function from, for Instance, takes a user 
datatype to a fixed point of a polyp blfunctor. 

This deep encoding also requires the conversion functions to be directly recursive. As 
an example, we show how to encode the standard list type: 

instance PolyP [] where 
type PF [] = U :+: (P :x: I) 

from [] = In (L U) 

from (h : t) = In (R (P h :x: I (from t))) 

to (In (L U)) = [] 

to (In (R (P h :x: 1 1))) = h : to t 

To see an example with composition, consider the type of rose trees: 

data Rose p — Rose p [ Rose p ] 

Its representation In polyp follows: 

instance PolyP Rose where 
type PF Rose = P :x: (PF [] :o: I) 

from (Rose x I) = In (P x :x: Comp (pmap (I o from) (from I))) 
to (In (P x :x: Comp I)) = Rose x (to (pmap (to o unl) I)) 

Note that we use PF [] (and not []) In the pattern functor for Rose. Consequently we 
need to use pmap to map over the lists of rose trees, also relying on the fact that [] Is 
an Instance of PolyP. 

The polyp pmap function on lists corresponds to the standard Haskell map function. 
We can write a more user-friendly Interface to pmap to take care of the necessary 
conversions: 
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gfmap :: (PolyP 5, Bimap (PF 5)) => {a^fi)^5a^5fi 
gfmap f = to o pmap f o from 

We call this function gfmap as It Is the generic variant of the fmap function from the 
standard Functor class. As an example, gfmap (+1) [1,2, 3] evaluates to [2, 3, 4]. 

We have seen that polyp generalises regular to add support for one datatype pa- 
rameter. In the next chapter we continue to Increase the flexibility of generic program- 
ming approaches, this time by considering how to support families of mutually recursive 
datatypes. 
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CHAPTER 5 



The multirec library 



Like polyp, the multirec library Rodriguez Yakushev et al. 2009 is also a general- 
isation of regular, allowing for multiple recursive positions instead of only one. This 
means that families of mutually recursive datatypes can be encoded in multirec. While 
both regular and polyp deal with mutual recursion by encoding datatypes (other than 
that being defined) as constants, in multirec the family is defined as a single group, 
allowing for catamorphisms over families, for instance. 



5.1 Agda model 

In multirec, types are represented as higher-order (or indexed) functors: 

Indexed : Set -»• SeU 
Indexed a = a — > Set 

Codes themselves are parametrised over an index Set, that is used in the I case. Fur- 
thermore, we have a new code ! for tagging a code with a particular index: 

data Code (I : Set) : Set where 
U : Code I 

I : I Code I 

I : I Code I 

: (F G : Code I) ->■ Code I 
_®_ : (F G : Code I) -> Code I 

Since multirec does not support parameters, there is no code for composition. 
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The interpretation is parametrised by a function r that maps indices (recursive po- 
sitions) to Sets, and a specific index i that defines which particular position we are 
interested in; since a code can define several types, [_] is a function from the index of 
a particular type to its interpretation. For an occurrence of an index I we retrieve the 
associated set using the function r. Tagging constrains the interpretation to a particular 
index j, so its interpretation is index eguality: 



{I : Set} 



U 
lj 

'j 
F ( 
F ( 



G 
G 



-> Code I 

T 

rj 

i = j 

[F]riW I 
If Fl rix 1 



Indexed I 



Indexed I 



r i 
r i 



Mapping is entirely similar to the regular map, only that the function being mapped 
is now an index-preserving map. An index-preserving map is a transformation between 
indexed functors that does not change their index: 



{I : Set} - 
= VI -> ri 



Indexed I 
* s i 



Indexed 



Set 



The map function is then an index-preserving map over the interpretation of a code: 



map : {1 


: Set } { r s 


: Indexed 1} (F : Code 1) - 


map U 


f i tt 


= tt 


map (1 j) 


f i X 


= fjx 


map (! j) 


f i X 


= x 


map (F © 


G) f i (inji x) 


= inji (map F f i x) 


map (F © 


G) f i (inj 2 y) 


= inj 2 (map G f i y) 


map (F <g> 


G) f i (x , y) 


= map F f i x , map G f i y 



We also need an indexed variant of the fixed-point operator: 

data \i {I : Set} (F : Code I) (i : I) : Set where 

(_) : [ F ] („ F) i n F i 

To show an example involving mutually recursive types we encode a zig-zag seguence 
of even length. Consider first the family we wish to encode, inside a mutual block as 
the datatypes are mutually recursive: 

mutual 
data Zig : Set where 

zig : Zag ->• Zig 
end : Zig 

data Zag : Set where 

zag : Zig -» Zag 
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We can encode this family In multirec as follows: 

ZigC : Code (T W T) 
ZigC = I (inj 2 tt) © U 

ZagC : Code (T W T) 
ZagC = I (1^ tt) 

ZigZagC : Code (T W T) 

ZigZagC = (! (in^ tt) <g> ZigC) © (! (inj 2 tt) <g> ZagC) 
zigZagEnd : \i ZigZagC (inji tt) 

zigZagEnd = ( inji (refl , inj-i ( inj 2 (refl , ( inji (refl , inj 2 tt) )) }) } 

zigZagEnd encodes the value zig (zag end), as Its name suggests. Note how we define the 
code for each type in the family separately (ZigC and ZagC), and then a code ZigZagC for 
the family, encoding a tagged choice between the two types. As a conseguence, proofs 
of Index eguallty (witnessed by refl) are present throughout the encoded values. 



5.2 Haskell implementation 



The encoding of multirec In Haskell makes use of more advanced type features than 
are necessary for regular. Again we have a number of representation datatypes, now 
parametrised over an Indexed recursion point argument p and an Index i: 



data U 
data I k 
data K a 

data (</> :+ 



( 
( 

(P 
4>) (P 



data (<j) :x: <//) (p 



*) i = 

*) i = 

*) ; = 

★) ; = 

*) i = 



U 

I {unl :: p k} 
K {unK :: a} 
L(</>p,) | R(tPpi) 
</> p i :x: <// p i 



This Is a standard way of encoding mutual recursion as higher-order fixed points. The p 
argument Is used as a selector of which type to recurse Into; In the I case for recursive 
occurrences, It Is used with a specific index k. 

The i argument has a different purpose. Since we are encoding multiple datatypes In 
a single datatype, we use ( to focus on one particular type of the family. For focusing 
we use the > representation type, analogous to the ! code of the Agda model: 

data <j> > i ::(*—► *) — » * — » * where 

Tag :: 4> p ; -> (4> > l) p i 



We write > as a generalised algebraic datatype (GADT) |Schrljvers et al. 2009 to 
Introduce a type-level eguallty constraint (egulvalent to the _=_ Agda type), signalled 
by the appearance of i twice In the result type of Tag. This means that Tag can only be 
built (or type checked) when the reguested Index Is the same as that given as argument 
to >. The example datatype encoding In Section 5.2.2 makes the use of tagging more 
clear. 
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5.2.1 Generic functions 

As usual, generic functions are defined by giving Instances of a type class for each 
representation type. We show the higher-order mapping function for multirec: 

class H Functor £ 0 where 

hmap :: (forall i. £ i — > pi — ► at) — ► £ i — > 0 p i — ► </> a i 

Its type In Haskell Is similar to the type In Agda, transforming the recursive points of a 
representation type 0 from p to cr while keeping the Index i unchanged. Additionally, we 
take a function to apply In the recursive occurrences, similarly to regular, as multirec 
Implements a shallow view on data. Finally, the argument of type Z, i, which also 
occurs In the function for the recursive case, stands for the family Index. The Instance 
of HFunctor for I Illustrates Its use: 



class El £ i where 

proof :: £ i 



instance El £ i => HFunctor £ (I t) where 
hmapf_(lx) = I (f proof x) 



For a recursive occurrence under I we simply call the function we carry around. This 
function takes as first argument an Index for which type In the family It Is called on. To 
simplify the construction of such Indices we also define class El; an instance of El £ ( 
means that i is an Index of the family evidenced by the element proof of type £ i. In 
Section 5.2.2 we show how to Instantiate El for an example family, to further help clarify 



Its use. 

The remaining cases are simple; we recurse through sums, products, and tags, and 
return in units and constants: 



instance HFunctor £ U where 
hmap U = U 

instance HFunctor £ (K a) where 
hmap (K x) = Kx 

instance (HFunctor £ 0, HFunctor £ ip) => HFunctor £ (0 :+: 0) where 
hmap f p (L x) = L (hmap f p x) 
hmap f p (R x) = R (hmap f p x) 

instance (HFunctor £ 0, HFunctor £ 0) => HFunctor £ (0 :x: 0) where 
hmap f p (x :x: y) = hmap f p x :x: hmap f p y 

instance HFunctor £ 0 => HFunctor £ (0 > t) where 
hmap f p (Tag x) = Tag (hmap f p x) 



We can use hmap to define a higher-order catamorphlsm function, but we show first 
how to represent datatypes. 
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5.2.2 Datatype representation 

Families of datatypes are represented by Instances of the El class shown before, and 
also the Fam class: 

newtype l 0 a = l 0 a 

class Fam d; where 

type PF :: * — » *)::(* — > *) — > * — > * 

from <5 l — > t — » PF £ l 0 t 

to :: < — » PF ^ l 0 ( — > i 

In Fam we have the usual from and to conversion functions, and the pattern functor 
representation as a type family. The pattern functor takes the family £ as an argument, 
and Is Indexed over the recursive point argument (which we called p when defining the 
representation types) and a particular Index (. The conversion functions Instantiate p to 
lo, a type-level Identity. 

We revisit our zig-zag mutually recursive datatypes In Haskell to show an example 
Instantiation: 

data Zig = Zig Zag | End 
data Zag = Zag Zig 

The first step to encode these in multirec Is to define a datatype to represent the 
family: 

data ZigZag ( where 
ZigZagzig :: ZigZag Zig 
ZigZagzag " ZigZag Zag 

ZigZag now stands for the family type, with ZigZagzig and ZigZagzag being respectively 
the evidence for the presence of Zig and Zag In this family. This Is encoded In the 
Instance of El: 

instance El ZigZag Zig where 

proof = ZigZagzig 
instance El ZigZag Zag where 

proof = ZigZagzag 

We are left with the Instance of Fam: 

instance Fam ZigZag where 
type PF ZigZag = ((I Zag :+: U) > Zig) 
:+: (I Zig > Zag) 

from ZigZagzig (Zig zag) = L (Tag (L (I (l 0 zag)))) 
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from ZigZagzig End = L (Tag (R U)) 
from ZigZagzag (Zag zig) = R (Tag (I (l 0 zig))) 
to ZigZagzig (L (Tag (L (I (l 0 zag))))) = Zig zag 
to ZigZag^ (L (Tag (R U))) = End 

to ZigZagzag (R (Tag (I (l 0 zig)))) = Zag zig 

Since from and to operate on multiple datatypes we use their first argument, the family 
proof, to set which type we are converting. Pattern matching on the proof is also what 
allows us to produce the Tag constructor, as its type reguires that the value being 
constructed matches the index declared in the tag representation type >. 

Using the Fam class we can define a higher-order catamorphism function that takes a 
user datatype, converts it to the generic representation, and applies an algebra recur- 
sively using hmap: 

type Algebra I, p — forall i. £ i —> (PF p i —> p i 

cata :: (Fam <f, HFunctor £ (PF £)) => Algebra £ p — > i — > t — > p i 
cata f p = f p o hmap (A p (l 0 x) -* cata f p x) p o from p 

Note that this catamorphism does not have to return a single type for a family. Its 
return type is p (, implying that different types can be returned for different family 
indices. A generic length function for a family has a single return type Int, but an identity 
catamorphism does not, for instance. More realistic examples include the evaluation of 
an abstract syntax tree, which is shown as an example in the source distribution of 
multireel]] 

We have seen that adding support for families of datatypes reguires lifting the indices 
from Sets to Set-indexed functions. The reader might wonder if something similar can be 
done to lift polyp's restriction to a single parameter. That has been investigated before 
| Hesse link 2009, with mixed results: while it is possible, the resulting Haskell library 



is hard to use, and there is no support for family composition. In the next chapter we 
look at a different approach that does allow code composition. 



^http : //hac kage . haskell . org/package/multirec 
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CHAPTER 6 



Indexed functors 



This chapter presents indexed, a powerful approach to generic programming In Agda 
using Indexed functors. Unlike the previous approaches we have seen, Its universe 
Incorporates fixed points, and also supports composition, Indexing, and Isomorphisms. 

We study this approch In more detail since It generalises all of the previous ap- 
proaches. We provide proofs for the functor laws {Section 6.2 , define a decldable 
equality function {Section 6.3) , and show a zipper for Indexed fun ctors {Section 6.4| . 
We have not shown these for the previous libraries, but we show In Chapter 8 how the 
previous libraries can use the facilities provided by indexed. 



6.1 A universe for Indexed functors 



The indexed approach Is centered on the notion of an Indexed functor. A normal functor 
Is a type constructor, I.e. of Agda type Set — > Set. An Indexed functor allows an 
Indexed set both as Input and as output, where an Indexed set Is a function mapping 
a concrete "Index" type to Set. Similarly to multirec, families of mutually recursive 
datatypes can be expressed as a single Indexed functor, by choosing appropriate Indices. 
Parametrised datatypes are represented using Indices as well; In this way recursion and 
parametrlsatlon are treated uniformly, allowing for flexible composition. Similarly to the 
universes of Altenklrch et al. 2007 , indexed Includes the fixed-point operator within the 
universe, which simplifies code reuse. To allow for easy encapsulation of user-defined 
datatypes, datatype Isomorphisms are part of the universe Itself. 

The result Is a very general universe that can encode many more datatypes than the 
previous approaches we have seen. Unlike Chapman et al. 2010, however, indexed Is 
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not a minimal, self-encoding universe, but instead a representation that maps naturally 
to the usual way of defining a datatype. 

Unlike the libraries we have seen so far, the indexed approach was originally de- 
scribed in Agda Lbh and Magalhaes 2011 . An added advantage of working in a 



dependently typed setting is that properties of generic functions are just generic func- 
tions themselves, and can be proven within the same framework. We show how indexed 
functors adhere to the functor laws. As examples of how to work with the universe, we 
derive the catamorphism and other recursion schemes. Along with all the functionality 
that can be defined using the recursive morphisms, we show that our approach also 
allows defining functions by direct induction on the codes, and show decidable eguality 
as an example. Finally, we present the basic ingredients for navigation of datatypes 
described as indexed functors using a zipper Huet 1997, relying on an underlying 
derivative |McBride" 2001 for the universe. 



6.1 .1 Basic codes 

The codes of the universe describe indexed functors, and are therefore parametrised over 
two sets. Intuitively, these can be thought of as the indices for inputs and outputs of the 
type described by the code, so we generally call the first argument I and the second O. 
A code I ► O describes a type that is parametrised over inputs from I and that is itself 
indexed over O. In many standard examples, I and O will be instantiated to finite types. 
We get a classic functor of type Set —> Set back by instantiating both I and O with the 
one-element type T. In case we want a code to represent a family of n datatypes, we 
can instantiate O to the type Fin n that has n elements. 

Since the indexed universe is large, we present it in an incremental fashion, starting 
with the base cases: 



data _>_ (I : Set) (O 
Z : I ► O 
U : I ► O 



: Set) : Seti where 



The codes are a relation between two sets, so we use an infix operator for the universe 
type, taking the input set on the left and the output set on the right. The constructor 
Z is used for empty types, and U for unit types. Both base codes are polymorphic on 
the input and output indices. We do not use a code K for inclusion of arbitrary constant 
types, as these prove troublesome when defining generic functions. Instead we use 
isomorphisms, as shown later in this chapter. 

Disjoint sum, product and composition are used for combining codes: 



: I^O->l*-0->l*-0 
_®_ : l*-0->l*-0->l*-0 
: {M: Set} M ► O I ► M -> I ► O 

Sum (_©_) and product (_®_) combine two codes with input I and output O to produce a 
code I ► O. For a composition F © G we reguire the codes to combine to be compatible: 



44 



6.1 A universe for indexed functors 



the output of code G needs to be the same as the input of the code F (namely M). We 
connect the output of G to the input of F, and produce a code with the input of G and 
the output of F. 

Together with the universe, we define an interpretation function that establishes the 
relation between codes and Agda types. A code I ► O is interpreted as a function 
between indexed functors I > O: 

Indexed : Set -»- Seti 
Indexed I = I — > Set 

_t>_ : Set ->■ Set ->■ Seti 

I > O = Indexed I — > Indexed O 

When defining the actual interpretation function [_J, we thus have a parameter r : 
I — > Set describing all the input indices and a parameter o : O selecting a particular 
output index available: 

[_] : {I O : Set} — ► I ► O -» I > O 

[Z Jro = ± 

[U ] ro = T 

[FffiG] ro = IF] row [G] ro 

[F®G] ro = IF] ro x [G] ro 

[F@G]ro = (!F]olG])ro 

We interpret Z as the empty type _!_ (with no constructors), and U as the singleton type 
T. Each of the three constructors for sum, product, and composition is interpreted as 
the corresponding concept on Agda types. 

With this part of the universe in place we can already encode some simple, non- 
recursive datatypes. Consider the type of Boolean values Bool with constructors true 
and false: 

data Bool : Set where 
true : Bool 
false : Bool 

It is a single, non-indexed datatype, which takes no parameters and is not recursive. 
We can encode it as an indexed functor using a type with zero inhabitants for I and a 
type with one inhabitant for O: 

'BoolC : 1 ► T 
'BoolC = U © U 

To convert between the type of Booleans and its representation we use two functions: 

fromBool : Bool ->• I 'BoolC ] (A ()) tt 
fromBool true = inj-i tt 
fromBool false = inj 2 tt 
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toBool : I 'BoolC ] (A ()) tt Bool 
toBool (inji tt) = true 
toBool (inj 2 tt) = false 

We instantiate the input indexed set to (A ()), the absurd match. Recall that its type 
is Indexed _L, or _L —> Set. A function that takes an uninhabited type as an argument 
will never be called, so to define it we simply have to match on the empty type with the 
absurd pattern (). The output index is set to tt, as this family only has a single output 
index. 



6.1.2 Isomorphisms 



Being in a dependently-typed language, we can also provide the proof that fromBool 
and toBool indeed form an isomorphism: 

isoBooh : V b — > toBool (fromBool b) = b 
isoBooh true = refl 
isoBooh false = refl 

isoBooh : V b —> fromBool (toBool b) = b 
isoBool 2 (inji tt) = refl 
isoBooh (inj 2 tt) = refl 

For convenience, we wrap all the above in a single record, which we introduced in 
Section 2.2.3 but repeat here for convenience: 



infix 3 

record ~ 
field 

from 

to 

isoi 

iS02 



(A B : Set) : Set where 



A - 
B - 
Vx 
Vx 



B 
A 



to (from x) 
from (to x) 



The particular instantiation for Bool simply uses the functions we have defined before: 



isoBool : (r 
isoBool r o = 



Indexed _L) (o 
record {from 
; to 
; isoi 
; iso2 



T) ->• Bool 
= fromBool 
= toBool 
= isoBooh 
= isoBoo^} 



'BoolC 1 r o 



Isomorphisms are useful when we want to view a type as a different, but eguivalent 
type. In the case for Bool and J 'BoolC J, for instance, the former is more convenient 
to use, but the latter can be used generically. If we add isomorphisms to the universe, 
generic functions can automatically convert between a user type and its representation: 
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Iso : (C : I ► O) — ► (D : I > O) — » 

((r : Indexed I) ->• (o : O) -> Dro~[[C]ro) -> I ► O 

The code Iso captures an isomorphism between the interpretation of a code C and an 
indexed functor D. This indexed functor D is used in the interpretation of the code Iso: 

[ Iso C D e ] r o = D r o 

Having Iso has two advantages. First, it enables having actual user-defined Agda 
datatypes in the image of the interpretation function. For example, we can give a 
code for the Agda datatype Bool now: 

'Bool' : _L ► T 

'Bool' = Iso 'BoolC (A _ _ -» Bool) isoBool 

Secondly, Iso makes it possible to reuse previously defined types inside more complex 
definitions, while preserving the original code for further generic operations — in |Sec- 



tion 6.1.7 we show an example of this technigue by reusing the code for natural numbers 
inside the definition of a simple language. 

Note that the Iso code is not specific to indexed only; the previous Agda models could 
have a code for isomorphisms as well. But since this code does not add expressiveness 
to the universe, and the models are a concise description of each library, we have not 
added isomorphisms to them. 



6.1.3 Adding fixed points 

To encode recursive datatypes we need some form of fixed-point operator. In the Agda 
models we have seen so far this operator is a separate datatype, outside the universe. 
Indexed functors, however, have the nice property of being closed under fixed points: if 
we have an indexed functor where the recursive calls come from the same index set as 
the output, i.e. a functor O > O, then its fixed point will be of type _L > O. If we consider 
a functor with parameters of type I W O > O, with input indices being either parameters 
from I, or recursive calls from O, we can obtain a fixed point of type I > O. 

Since the fixed point of an indexed functor is itself an indexed functor, it is convenient 
to just add fixed points as another constructor to the type of codes: 

Fix : (I l±l O) ► O — > I ► O 

As indicated before, the code Fix transforms a code with I l±l O input indices and O output 
indices into a code of type I ► O. Naturally, we still need a way to actually access 
inputs. For this we use the I code, similarly to the previous approaches: 

I : I -> I ► O 

Using I we can select a particular input index (which, in the light of Fix, might be either 
a parameter or a recursive call). 



47 



6 Indexed functors 



We use a datatype ^/ with a single constructor (_) to generate the interpretation 
of Fix F from that of F. We know that F : (I W O) ► O, so the first argument to [ F ] 
needs to discriminate between parameters and recursive occurrences. We use r | \i F r 
for this purpose, i.e. for parameters we use r : I —> Set, whereas recursive occurrences 
are interpreted with \i F r : O — > Set: 

_|_ : {I J : Set} ->■ Indexed I ->■ Indexed J ->■ Indexed (I W J) 
(r | s) (inj! i) = r i 
(r | s) (inj 2 j) = sj 

mutual 

data ^{10 : Set} (F : (I W O) ► 0) (r : Indexed I) (o : O) : Set where 

(_) : [ F J (r | v F r) o v F r o 

[ Fix F ] r o = f.i F r o 
II i] ro = ri 

The interpretation of I uniformly invokes r for every input index i. 

As an example of a datatype with parameters and recursion, we show how to encode 
parametric lists (introduced in |Section ZT . We start by encoding the base functor of 
lists: 

'ListC : (T W T) ► T 

'ListC = U © (I (inj-i tt) ® I (inj 2 tt)) 

The arguments of _►_ reflect that we are defining one type (output index T) with two 
inputs (input index T (±1 T), where one represents the parameter and the other represents 
the recursive call. 

We use a product to encode the arguments to the _::_ constructor. The first argument is 
an occurrence of the first (and only) parameter, and the second is a recursive occurrence 
of the first (and only) type being defined (namely list). 

Using Fix, we can now close the recursive gap in the representation of lists: 

'ListF' : T ► T 
'ListF' = Fix 'ListC 

We can confirm that our representation is isomorphic to the original type by providing 
conversion functions: 



fromList : V{ro} -> [ro] -> J 'ListF' ] r o 

fromList [] = ( inj-| tt } 

fromList (x :: xs) = ( inj 2 (x , fromList xs) } 

toList : V {ro} -> [['ListF'] ro ->■ [ro] 

toList ( inj! tt } = [] 

toList ( inj 2 (x , xs) } = x :: toList xs 
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As before, we can show that the conversion functions really form an isomorphism. The 
isomorphism is entirely trivial, but the implementation is a bit cumbersome because we 
have to explicitly pass some implicit arguments in order to satisfy Agda's typechecker: 



toList {r} (fromList 



I 



x) (isoListi {r} t) 
-> fromList (toList I) = I 

inj 2 (h , x) }) (isoList2 t) 



isoListi : V {r 0} -> (I : [ r 0]) 
isoListi = refl 

isoListi {r} (h :: t) = cong (Ax -> h :: 

isoList 2 : V{ro} -> (I : [ 'ListF' ] r 0) 
isoList2 ( inji tt } = refl 
isoList2 ( inj 2 (h , t) } = cong (Ax -> { 

isoList : V {ro} [ro] ~ ['ListF'] ro 

isoList { r} = record {from = fromList 

; to = toList 

; isoi = isoListi {r} 

; iso 2 = isoList 2 } 



For conciseness we will refrain from showing further proofs of isomorphisms. 
We now define a code for lists including the isomorphism between [_] and ^/ 'ListC: 



'List' : T ► T 

'List' = Iso (Fix 'ListC') (A f t 



[f t]) (A r 0 ->• isoList) 



This code makes it possible for us to use actual Agda lists in generic operations, and 
explicit applications of the conversion functions toList and fromList are no longer neces- 
sary. 

Having Fix in the universe (as opposed to using it externally) has the advantage that 
codes become more reusable. For example, we can reuse the code for lists we have just 
defined when defining a code for rose trees {Section 6.1 ,6| . This is an instance of the 
common situation where a fixed point is used as the first argument of a composition. 
Therefore indexed allows encoding datatypes that involve several applications of Fix. 



6.1.4 Mapping indexed functors 

To show that indexed codes are interpreted as indexed functors, we define a map 
operation on codes. 

Since we are working with indexed sets rather than sets, we have to look at arrows 
between indexed sets, which are index-preserving mappings: 

_=4_ : {I : Set} -> Indexed I Indexed I Set 
r=£s = Vi— >ri— >si 

As in multirec, map lifts such an index-preserving function r =4 s between two indexed 
sets r and s to an indexed-preserving function [C]r=?[Cjson the interpretation of 
a code C: 
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map : {10 : Set} {rs : Indexed 1} — > 

(C:I^O)->r=U->[Cjr=|[Cjs 

Note that Lf we choose I = O = T, the Indexed sets become isomorphic to sets, and 
the index-preserving functions become isomorphic to functions between two sets, i.e. we 
specialise to the well-known Haskell-like setting of functors of type Set — > Set. 
Let us look at the implementation of map: 

map Z f o () 
map U f o tt = tt 
map (I i) f o x = fix 

If we expand the type of map we see that it takes four explicit arguments. The first 
two are the code and the index-preserving function f. The type of the function returned 
by map can be expanded toVo —> JCjro — > JCjso, explaining the remaining two 
arguments: an arbitrary output index o, and an element of the interpreted code. 

The interpretation of code Z has no inhabitants, so we do not have to give an im- 
plementation. For U, we receive a value of type T, which must be tt, and in particular 
does not contain any elements, so we return tt unchanged. On I we get an element x 
corresponding to input index i, which we supply to the function f at index i. 

For sums and products we keep the structure, pushing the map inside. Composition 
is handled with a nested map, and for fixed points we adapt the function argument to 
take into account the two different types of indices; using an auxiliary _||_ operator to 
merge index-preserving mappings, left-tagged indices (parameters) are mapped with f, 
whereas right-tagged indices (recursive occurrences) are mapped recursively: 

map (F © G) f o (inji x) = inji (map F f o x) 

map (F © G) f o (inj 2 x) = inj 2 (map G f o x) 

map (F ® G) f o (x , y) = map F f o x , map G f o y 

map (F © G) f o x = map F (map G f) o x 

map (Fix F) f o ( x ) = ( map F (f || map (Fix F) f) o x } 

The operator for merging index-preserving mappings, used in the fixed point case, fol- 
lows: 



_||_ : {I J : Set}{ru : Indexed I} {sv : Indexed J} ->• 

r =4 u -» s =4 v -> (r | s) =? (u | v) 

(f || g) (inj! x) = f x 
(f || g) (inj 2 x) = g x 

We are left with a case for isomorphisms to complete our definition of map, which we 
define with judicious use of the conversion functions: 

map {r = r} {s = s} (Iso C D e) f o x with (e r o , e s o) 

... | epi , ep 2 = _~_.to ep 2 (map C f o (_~_.from epi x)) 
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We use Agda's with construct to pattern-match on the embedding-projection pairs, from 
which we extract the necessary conversion functions. 

As an example, let us look at the instance of map on lists. We obtain it by specialising 
the types: 

mapList : {A B : Set} -> (A B) [A] -> [B] 
mapList f = map 'List' (const f) tt 

We are in the situation described before, where both index sets are instantiated to T. 
Note that we do not need to apply any conversion functions, since 'List' contains the 
isomorphism between lists and their representation. We discuss how to prove the functor 
laws for map in |Section 6.2| 

For now, we can confirm that mapList works as expected on a simple example: 

mapListExample : mapList su (1 :: 2 :: []) = (2 :: 3 :: []) 
mapListExample = refl 

Since this code typechecks, we know that mapList su (1 :: 2 :: []) indeed evaluates to 
2:: 3-D. 

6.1.5 Recursion schemes 

Eguipped with a mapping function for indexed functors, we can define basic recursive 
morphisms in a conventional fashion: 

id^ : {I : Set} {r : Indexed I} — ^ r =^ r 
id^ i = id 

cata : {I O : Set} {r : Indexed I} {s : Indexed 0} -> 

(C : (I W O) ► O) -» [ C J (r | s) =4 s -» I Fix C ] r =» s 
cata C cp i ( x } = cp\ (map C (id=$ || cata C cp) i x) 

The catamorphism on indexed functors is not much different from the standard functorial 
catamorphism. The important difference is the handling of left and right indices: since 
we wish to traverse over the structure only, we apply the identity on indexed sets id^ 
to parameters, and recursively apply cata to right-tagged indices. 

As an example, let us consider lists again and see how the foldr function can be 
expressed in terms of the generic catamorphism: 

_V_ : {A B C : Set} — > (A — > C) — > (B — > C) — > (A l+l B) — > C 
(r V s) (inj t i) = r i 
(r V s) (inj 2 j) = sj 

foldr : {A B : Set} -^(A^B^B)^B^[A]^B 
foldr {A} c n I = cata {r = const A} 'ListC cp tt (fromList I) 
where cp = const (const n V uncurry c) 
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The function foldr invokes cata. The parameters are instantiated with const A, i.e. there 
is a single parameter and it is A, and the recursive slots are instantiated with const B, 
i.e. there is a single recursive position and it will be transformed into a B. We invoke 
the catamorphism on 'ListF', which means we have to manually apply the conversion 
fromList on the final argument to get from the user-defined list type to the isomorphic 
structural representation. Ultimately we are interested in the single output index tt that 
lists provide. 

This leaves the algebra cp. The generic catamorphism expects an argument of type 
['ListF' ] (r | s) =4 s, which in this context reduces to (i : T) — > T W (A x B) — > B. 
On the other hand, foldr takes the nil- and cons- components separately, which we can 
join together with _V_ to obtain something of type T l+l (A x B) —> B. We use const to 
ignore the trivial index. 

We can now define the length of a list using foldr and check that it works as expected: 

length : {A : Set} — > [A] — > N 
length = foldr (const su) ze 

lengthExample : length (1 :: 0 :: []) = 2 
lengthExample = refl 

Many other recursive patterns can be defined similarly. We show the definitions of 
anamorphism and hylomorphism: 

ana : {I O : Set} {r : Indexed I} {s : Indexed 0} 

(C : (I W O) ► O) -> s =t I C ] (r | s) -> s =t I Fix C ] r 
ana C tp i x = ( map C (id^ || ana C tp) i (tp i x) ) 

hylo : {10 : Set} {r : Indexed I} {st : Indexed 0} 

(C : (I l±l O) ► O) -> I C ] (r 1 1) =t t -> s =t [ C ] (r | s) s =* t 

hylo C <p tp i x = cp i (map C (id=$ || hylo C cp tp) i (tp i x)) 
6.1.6 Using composition 

To show how to use composition we encode the type of rose trees: 

data Rose (A : Set) : Set where 
fork : A — > [Rose A] — > Rose A 

The second argument to fork, of type [Rose A], is encoded using composition: 

'RoseC : (T W T) ► T 

'RoseC = I (inj! tt) ® ('ListF ® I (inj 2 tt)) 

'RoseF' : T ► T 
'RoseF' = Fix 'RoseC 
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Note that the first argument of composition here is 'ListF', not 'ListC. We thus really 
make use of the fact that fixed points are part of the universe and can appear anywhere 
within a code. We also show that codes can be reused in the definitions of other codes. 
The conversion functions for rose trees follow: 

fromRose : {r : Indexed T} {o : T} -> Rose(ro) -* [ 'RoseF' ] ro 
fromRose (fork x xs) = ( x , map 'ListF' (A i — > fromRose) tt (fromList xs) } 

toRose : {r : Indexed T} {o : T} -* [['RoseF'] ro — > Rose (r o) 
toRose ( x , xs } = fork x (toList (map 'ListF' (A i -* toRose) tt xs)) 

The use of composition in the code implies the use of map in the conversion functions, 
since we have to map the conversion over the elements of the list. This also means that 
to provide the isomorphism proofs for Rose we need to have proofs for the behavior of 
map. We describe these in detail in Section 6.2 



6.1.7 Parametrised families of datatypes 

As an example of the full power of abstraction of indexed functors we show the represen- 
tation of a family of mutually recursive parametrised datatypes. Our family represents 
the Abstract Syntax Tree (AST) of a simple language: 

mutual 

data Expr (A : Set) : Set where 

econst : N — > Expr A 

add : Expr A — > Expr A — > Expr A 

evar : A — » Expr A 

elet : Decl A -> Expr A -> Expr A 

data Decl (A : Set) : Set where 

assign : A — > Expr A — > Decl A 
seq : Decl A -> Decl A -> Decl A 

In our AST, an expression can be either a natural number constant, the addition of two 
expressions, a variable, or a let declaration. Declarations are either an assignment of 
an expression to a variable, or a pair of declarations. 

We can easily encode each of the datatypes as indexed functors. We start by defining 
a type synonym for the output indices, for convenience: 

AST : Set 
AST = T W T 

expr : AST 
expr = inj-| tt 

decl : AST 
decl = inj 2 tt 

Since we are defining two datatypes, we use a type with two inhabitants, namely 
T l+l T. Note that any other two-element type such as Bool or Fin 2 would also do. We 
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define expr and decl as shorthands for each of the indices. We can now encode each of 
the types: 

'ExprC : (T W AST) ► AST 
'ExprC = ? 0 

© I (inj 2 expr) ® I (inj 2 expr) 

© I (inji tt) 

© I (inj 2 decl) ® I (inj 2 expr) 

'DecIC : (T W AST) ► AST 
'DecIC = I (inj-i tt) ® I (inj 2 expr) 
© I (inj 2 decl) <g> I (inj 2 decl) 

Our codes have type (T W AST) ► AST, since we have one parameter (the type of 
the variables) and two datatypes in the family. For expressions we want to reuse a 
previously defined W code for N (which we do not show). However, W has type _L ► T, 
which is not compatible with the current code, so we cannot simply enter W in the ?o 
hole. 

We need some form of re-indexing operation to plug indexed functors within each 
other. Therefore we add the following operation to our universe: 

_?_\_ : {I' O' : Set} -> I' ► O' -> (I' -> I) -> (O ->• O') -> I ► O 

Now we can fill the hole ? 0 with the expression 'N' f (A ()) \, const tt. The interpretation 
of this new code is relatively simple. For the input, we compose the re-indexing function 
with r, and for the output we apply the function to the output index: 

[F/fVg]ro = [F](rof)(go) 

Mapping over a re-indexed code is also straightforward: 

map (F T g V h) f o x = map F (f o g) (h o) x 

Finally, we can join the two codes for expressions and declarations into a single code 
for the whole family. For this we need an additional code to specify that we are defining 
one particular output index, similarly to multirec: 

! : O I ► O 

The code ! is parametrised by a particular output index. Its interpretation introduces 
the constraint that the argument index should be the same as the output index we select 
when interpreting: 

[ ! o' ] r o = 0 = 0' 

Its usefulness becomes clear when combining the codes for the AST family: 

'ASTC : (T W AST) ► AST 
'ASTC = ! expr ® 'ExprC 
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© ! decl ® 'DecIC 

'ASTF' : T ► AST 
ASTF' = Fix ASTC 

In ASTC (the code for the family before closing the fixed point) we encode either an 
expression or a declaration, each coupled with an equality proof that forces the output 
index to match the datatype we are defining. If we now select the expr index from the 
interpretation of ASTF', then ! decl yields an uninhabited type, whereas ! expr yields a 
trivial equality, thereby ensuring that only 'ExprF' corresponds to the structure in this 
case. If we select decl in turn, then only 'DecIF' contributes to the structure. 

We can now define the conversion functions between the original datatypes and the 
representation: 

mutual 

toExpr : {r : T ->• Set} ->• [ASTF'] r expr -> Expr(rtt) 

toExpr ( inj-| (refl , inji x) ) = econst x 

toExpr { inj-| (refl , inj 2 (in^ (x , y))) ) = add (toExpr x) (toExpr y) 

toExpr ( inji (refl , inj 2 (inj 2 (in^ x))) ) = evar x 

toExpr ( inj-i (refl , inj 2 (inj 2 (inj 2 (d , e)))) } = elet (toDecl d) (toExpr e) 

toExpr ( inj 2 (() , _) } 

toDecl : {r : T -> Set} -> [ASTF'] r decl ->■ Decl(rtt) 
toDecl ( inj-, (() , _) } 

toDecl ( inj 2 (refl , inji (x , e)) } = assign x (toExpr e) 
toDecl ( inj 2 (refl , inj 2 (di , d 2 )) } = seq (toDecl di) (toDecl d 2 ) 

The important difference from the previous examples is that now we have absurd patterns. 
For instance, in toExpr we have to produce an Expr, so the generic value starting with 
inj 2 is impossible, since there is no inhabitant of the type expr = decl. Dually, in the 
conversion from the original type into the generic type, these proofs have to be supplied: 

mutual 

fromExpr : {r : T -> Set} ->■ Expr(rtt) -> [ASTF'] r expr 

fromExpr (econst x) = ( inj! (refl , inj! x) ) 

fromExpr (add x y) = ( inji (refl , inj 2 (inj! (fromExpr x , fromExpr y))) } 

fromExpr (evar x) = ( inji (refl , inj 2 (inj 2 (inj! x))) ) 

fromExpr (elet d e) = ( inji (refl , inj 2 (inj 2 (inj 2 (fromDecl d , fromExpr e)))) } 

fromDecl : {r : T -* Set} -> Decl(rtt) ->• [ASTF'] rdecl 

fromDecl (assign xe) = ( inj 2 (refl , inji (x , fromExpr e)) } 

fromDecl (seq di d 2 ) = ( inj 2 (refl , inj 2 (fromDecl di , fromDecl d 2 )) } 

At this stage the proofs are trivial to produce (refl), since we know exactly the type of 
the index. 
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6.1.8 Arbitrarily indexed datatypes 

The index types of a functor need not be finite types. Consider the following datatype 
describing lists of a fixed length (vectors): 

infixr 5 

data Vec (A : Set) : N ->• Set where 

[] Vec A ze 

: { n : N} -> A -> Vec A n -> Vec A (su n) 

The type Vec is indexed by the type of natural numbers N. In fact, for a given type A, 
Vec A defines a family of sets: Vec A ze (which contains only the empty list), Vec A (su ze) 
(all possible singleton lists), and so on. As such, we can see Vec as a code with one 
input parameter (the type A) and N output parameters: 

'VecC : N ->• (T W N) ► N 
'VecC ze = ! ze 

'VecC (su n) = ! (su n) <g> I (iriji tt) ® I (inj 2 n) 

Note, however, that we need to parameterise 'VecC by a natural number, since the code 
depends on the particular value of the index. In particular, a vector of length su n is 
an element together with a vector of length n. Unlike in the AST example, we cannot 
simply sum up the codes for all the different choices of output index, because there are 
infinitely many of them. Therefore, we need yet another code in our universe: 

E : {C : ± ► T} ->• ([ C ] (const T) tt I ► O) I ► O 

The code E introduces an existential datatype: 

data 3 {A : Set} (B : A -> Set) : Set where 
some : V{x} — > Bx — > 3 B 

[Efjro = 3(Ai Ifilro) 

Note that E is parametrised by a function f that takes values to codes. Arguments of 
f are supposed to be indices, but we would like them to be described by codes again, 
since that makes it easier to define generic functions over the universe. Therefore, we 
make a compromise and choose a code for a single unparametrised datatype _L ► T 
rather than an arbitrary Set — for more discussion, see |Section 6.1.10] 

To create a value of type [ E f ] we need a specific witness i to obtain a code from f. 
When using a value of type [ E f ] we can access the index stored in the existential. 

Here is the map function for E: 

map (E g) f o (some { i } x) = some (map (g i) f o x) 
Using E, we can finalise the encoding of Vec: 
'VecF' : T ► N 

'VecF' = Fix(E{C = 'N'} 'VecC') 
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We make use of the fact that we already have a code 'W for our index type of natural 
numbers. 

Finally we can provide the conversion functions. We need to pattern-match on the 
implicit natural number to be able to provide it as existential evidence in fromVec', and 
to be able to produce the right constructor in toVec': 

fromVec : V {n r} -> Vec (r tt) n ->■ [ 'VecF' ] r n 

fromVec{n = ze} = ( some {x = ze} refl ) 

fromVec {n = su m} (h :: t) = ( some {x = su m} (refl , (h , fromVec t)) } 

toVec : V {n r} -> [ 'VecF' ] r n -> Vec (r tt) n 

to Vec ( some {ze} refl) = [ 

to Vec { some {sun} (refl , (h , t)) } = h :: toVec t 



6.1.9 Nested datatypes 



Nested datatypes |Bird and Meerte"ns 1998 can be encoded in Agda using indexed 



datatypes. Consider the type of perfectly balanced binary trees: 

data Perfect (A : Set) : { n : N } -> Set where 

split : {n : N} -> Perfect A {n} x Perfect A {n} -> Perfect A { su n } 
leaf : A -> Perfect A { ze } 

Perfect trees are indexed over the naturals. A perfect tree is either a leaf, which has 
depth ze, or a split-node, which has depth su n and contains two subtrees of depth n 
each. 

In Haskell, this type is typically encoded by changing the parameters of the type in 
the return type of the constructors: split would have return type Perfect (Pair A), for some 
suitable Pair type. We can define Perfect' as such a nested datatype in Agda, too: 

data Pair (A : Set) : Set where 
pair : A -> A -> Pair A 



data Perfect' 
split : {A 
leaf : {A 



Set -» Seti where 

Set} -> Perfect' (Pair A) -> Perfect' A 
Set} -> A ->■ Perfect' A 



Now, Perfect' A is isomorphic to a dependent pair of a natural number n and an element 
of Perfect A {n}. 

We can therefore reduce the problem of encoding Perfect' to the problem of encod- 
ing Perfect, which in turn can be done similarly to the encoding of vectors shown in the 
previous section: 

'PerfectC : N — > (T W N) ► N 
'PerfectC (ze) = ! ze ® I (inji tt) 
'PerfectC (su n) = ! (su n) ® I (inj 2 n) ® I (inj 2 n) 
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'PerfectF' : T ► N 

'PerfectF' = Fix (E {C = W} 'PerfectC') 
We omit the embeddi.ng-projecti.on pair as Lt provides no new insights. 

6.1.10 Summary and discussion 

At this point we are at the end of showing how various sorts of datatypes can be 
encoded in the indexed universe. For reference, we now show the entire universe, its 
interpretation, and the map function: 



data 

Z 

u 



Fix 

E 
Iso 



(I : Set) (O : Set) : Seti where 
I ► O 
I ► O 

I -> I ► O 
O -> I ► o 

I ► O-»l*-O-H*-0 
I ►0->l*0->l*0 
V{M} -» M ► O -»• I ► M ->■ I ► O 

(i y o) ► o — » i ► o 

v {r o'} -»■ r ► c (r -> i) -» (o 

{C : _L ► T} -> ([C] (const T) tt -> 

(C : I ► O) -» (D : I > O) -> 

((r : Indexed I) ->• (o : O) D r o ~ 



- O") 
I ► O) 



ro 



I ► O 
I ► O 

I ► O 



data \i {I O : Set} (F : (I l+i O) ► O) (r : Indexed I) (o : O) : Set where 
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man 7 


f n n 




map U 
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— X 


cYl <^y\ 1 1 i 1 

nidp ij 
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— I I X 


map (F 0 G) 


f o (inii x^ 


= inji (map F f o x) 


map (F © G) 


f o (inj2 x) 


= inj 2 (map G f o x) 


map (F ® G) 


f o (x , y) 


= map F f o x , map G f o y 


map (F / g V 


h)fox 


= map F (f o g) (h o) x 


map (F © G) 


f 0 X 


= map F (map G f) o x 


map (! o') 


f 0 X 


= X 


map (I g) 


f o (some { i } x) 


= some (map (g i) f o x) 


map (Fix F) 


fo(x) 


= ( map F (f || map (Fix F) f) 


map {r = r}{s = s} (IsoCDe)fox with (e r o , e s o) 



... | (ep! , ep 2 ) = to ep 2 (map C f o (from ep! x)) 

where open _~_ 

Naturally, there are several variations possible in this approach, and the universe we 
have shown is not the only useful spot in the design space. We will now briefly discuss 
a number of choices we have taken. 

Perhaps most notably, we have avoided the inclusion of arbitrary constants of the form 

K : Set -> I ► O 

There are two main reasons why we might want to have constants in universes. One is 
to refer to user-defined datatypes; we can do this via Iso, as long as the user-defined 
datatypes can be isomorphically represented by a code. The other reason is to be able 
to include abstract base types (say, floating point numbers), for which it is difficult to 
give a structural representation. 

Adding constants, however, introduces problems as well. While map is trivial to 
define for constants — they are just ignored — most other functions, such as e.g. decidable 
eguality, become impossible to define in the presence of arbitrary constants. Additional 
assumptions (such as that the constants being used admit decidable eguality themselves) 
must usually be made, and it is impossible to predict in advance all the constraints 
necessary when defining a K code. 

A similar problem guides our rather pragmatic choice when defining Z. There are at 
least two other potential definitions for Z: 

Ii : Set -s- I ► O 

Z 2 : V{l'0'r'o'} {C : I' ► O'} -> ([ C ] r' o' -> I' ► O') -* I ► O 
In the first case we allow an arbitrary Set as index type. This, however, leads to problems 



with decidable eguality Morris 2007 Section 3.3], because in order to compare two 
existential pairs for eguality we have to compare the indices. Restricting the indices to 
representable types guarantees we can easily compare them. The second variant is more 
general than our current Z, abstracting from a code with any input and output indices. 
However, this makes the interpretation depend on additional parameters r' and o', which 
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we are unable to produce, in general. In our Z we avoid this problem by setting I' to _L 
and O' to T, so that r' is trivially A () and o' is tt. Our Z encodes indexing over a single 
unparametrised datatype. 

Note that the previous approaches we have seen can be obtained from indexed by 
instantiating the input and output indices appropriately. The regular library considers 
functors defining single datatypes without parameters. This corresponds to instantiating 
the input index to _L l±l T (no parameters, one recursive slot) and the output index to T, 
and allowing fixed points only on the outside, of type 1 y T ► T — > _L ► T. 

Adding one parameter brings us to the realm of polyp, with fixed points of bifunc- 
tors. The kind of the fixed-point operator in polyp corresponds exactly to the type 
T W T ► T — > T ► T, i.e. taking a bifunctor to a functor. Note also that polyp fur- 
thermore allows a limited form of composition where the left operand is a fixed point of a 
bifunctor (i.e. of type T ► T), and the right operand is a bifunctor (of type T W T ► T). 

Finally, multirec supports any finite number n of mutually recursive datatypes with- 
out parameters; that corresponds to fixed points of type _L l+l Fin n ► Fin n —> _L ► Fin n. 

The indexed approach thus generalises all of the previous libraries. It allows ar- 
bitrarily many mutually-recursive datatypes with arbitrarily many parameters, and it 
allows non-finite index types. We formalise this claim in |Chapter 8 

In the remainder of this chapter we provide further evidence of the usefulness of the 
indexed universe by showing a number of applications. 



6.2 Functor laws 



To be able to state the functor laws for the functorial map of |Section 6.1.4 we must 
first present some auxiliary definitions for composition and eguality of index-preserving 
mappings: 

_ °=t _ : { I : Set } { r s t : Indexed l}->s=M-»r=?s->r=M 
(f g) i x = f i (g i x) 

infix 4 ==j 

_ ==t _ : V {I : Set} {r s : Indexed I} (f g : r =4 s) -» Set 
f =^ g = Vix— >fix = gix 

We cannot use the standard propositional eguality directly in our laws because Agda's 
propositional eguality on functions amounts to intensional eguality, and we cannot prove 
our functions to be intensionally egual without postulating an extensionality axiom. 

We can now state the functor laws, which in the setting of indexed sets and index- 
preserving functions take the following form: 

map id : {I O : Set} {r : Indexed I} (C : I ► O) -> 
map {r = r} C id^ id^ 

map° : { I O : Set } { r s t : Indexed I } (C : I ► O) (f : s =4 t) (g : r =} s) ->• 
map C (f o-j g) =_j (map C f) o-^ (map C g) 
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Recall that id^ is the Identity mapping, Introduced In Section 6.1.5 



The first step to prove the laws Is to define a congruence lemma stating that mapping 
egulvalent functions results In egulvalent maps: 



map 

map v Z 
map v U 
map v (I i') 



{IO : 
(C : I 



Set} {rs 
► O) -> f ==, £ 

ipi() 

ip i x = refl 
ip i x = ip i' x 



Indexed I} {f g : 

=t g -* map C f 



r=^ s} 
==t map C g 



map (F } f V g) ip i x = map' F (ip o f) (g i) x 

For I we use the proof of eguallty of the functions. For re-lndexlng we need to adapt the 
Indices appropriately. Sums, products, and composition proceed recursively and rely on 
congruence of the constructors: 



map v (F 



G) 

map v (F © G) 
map v (F ® G) 
map v (F ® G) 
map v (! o') 



(inji x) 
(inj 2 x) 
(x,y) 

X 
X 



cong inji (map v F ip i x) 

cong inj 2 (map v G ip i x) 

cong 2 _,_ (map v F ip i x) (map v G ip i y) 

map v F (map v G ip) i x 

refl 



For Z, fixed points, and isomorphisms, the proof also proceeds recursively and by re- 
sorting to congruence of eguallty where necessary: 

map v (E g) ip i (some x) = cong some (map v (g _) ip i x) 

map v (Fix F) ip i ( x } = cong (_) (map v F (||-cong ip (map v (Fix F) ip)) i x) 

map v {r = r}{s = s} (Iso C D e) ip i x = 

cong (to (e s i)) (map v C ip i (from (e r i) x)) where open _~_ 

Note the use of an underscore In an expression In the case for E; this just means that 
Agda can automatically Infer the only possible argument for that position. We also open 
the _~_ record so that we can use Its operations unguallfied. 

For fixed points we use a lemma regarding congruence of the J|_ operator: 

||-cong : {I J : Set} {ru : Indexed I} {sv : Indexed J} 
{f-i f 2 : r =4 u} {gi g 2 : s =4 v} -> 
f-, =^ f 2 -» g-, =^ g 2 -» f-, || g-, =^ f 2 || g 2 

||-cong if ig (inji i) x = if i x 
ll-cong if ig (inj 2 i) x = ig i x 

We are now able to prove the functor laws. We start with map ld : 

map id : {I O : Set} {r : Indexed I} (C : I ► O) -> 
map {r = r} C id={ id^ 



61 



6 Indexed functors 

map id Z i () 

map' d U i x = refl 

map id (I i') i x = refl 

map id (F J f \ g) i x = map id F (g i) x 

The cases for Z, U, and I are trivial. For re-indexing we convert the output index and 
continue recursively; the input index is converted implicitly by Agda, filling in the implicit 
parameter r with r o f. Sums and products proceed recursively with congruence, while 
composition reguires transitivity as well: 

map' d (F © G) i (inji x) = cong inji (map' d F i x) 

map ld (F © G) i (inj 2 x) = cong inj 2 (map ld G i x) 

map id (F ® G) i (x , y) = cong 2 _,_ (map id F i x) (map id G i y) 

map id (F © G) i x = trans (map v F (map id G) i x) (map id F i x) 

Tagging obeys the law trivially, and for Z we proceed recursively with congruence on 
the some constructor: 

map' d (! o') i x = refl 

map' d (I g) i (some x) = cong some (map ld (g _) i x) 

The more complicated cases are those for fixed points and isomorphisms. For fixed points 
we first need an auxiliary lemma for eguality between identities over the _||_ operator, 
similar to ||-cong for congruence: 

1 1 -id : {I J : Set} {r : Indexed I} {s : Indexed J} {f : r =4 r} {g : s =J s} -> 

f =^ id^ ^ g ==; id^ -> (f || g) =^ id^ 

1 1 — id if ig (inji i) x = if i x 
1 1 -id if ig (inj 2 i) x = ig i x 

With this lemma we can prove the identity law for a fixed point. Recall that fixed 
points have left- and right-tagged codes. The map function proceeds recursively on the 
right (recursive occurrences), and it applies the mapping function directly on the left 
(parameters). So we have to show that both components are the identity transformation. 
We use the ||-id lemma for this, with a trivial proof for the left and a recursive call for 
the right: 

map id (Fix F) i ( x } = cong (_) (trans (map v F (||-id (A — > refl) 

(map id (Fix F))) i x) 

(map id F i x)) 

We are left with isomorphisms. The proof reguires only a careful use of symmetry and 
transitivity of propositional eguality: 

map ld {r = r} (Iso C D e) i x = sym (trans (sym ((iso! (e r i)) x)) 

(sym (cong (to (e r i)) 
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(map id C i (from (e r i) x))))) 
where open ~ 

Direct uses of sym and trans usually result in proof terms that are hard to write and 
understand; writing our proofs in equational reasoning style would be preferable. Fortu- 
nately, the standard library provides a module with convenient syntax for writing proofs 
in equational reasoning style: 

map id {r = r} (Iso C D e) i x = 
begin 

to (e r i) (map C id^ i (from (e r i) x)) 
=( cong (to (e r i)) (map ld C i (from (e r i) x)) } 

to (e r i) (from (e r i) x) 
=( isoi (e r i) x } 
I 

where open ~ 

This style makes it clear that the proof consists of two steps: removing the map C id^ 
application via a recursive call to the proof (of type map C id^ ==j id^), and using one 
of the isomorphism proofs. See the work of Mu et al. |2009 for a detailed account on 



this style of proofs in Agda. 

We have seen how to prove the identity functor law for the indexed map. The 
composition law map° is similar, so we omit it. 



6.3 Generic decidable equality 

Generic functions can be defined by instantiating standard recursion patterns such as 



the catamorphism defined in Section 6.1.5 or directly, as a type-indexed computation by 
pattern-matching on the universe codes. Here we show how to define a semi-decidable 
equality for our universe; in case the compared elements are equal, we return a proof 
of the equality. In case the elements are different we could return a proof of their 
difference; however, for conciseness, we only show the proof of equality. 
The type of decidable equality is: 

deqt : {I O : Set} {r : Indexed I} (C : I ► O) ->• SemiDec r -» SemiDec ([ C ] r) 

Note that to compute the equality SemiDec (J C ] r) we need the equality on the 
respective recursive indexed functors (SemiDec r). We use a type constructor SemiDec 
to define our type of semi-decidable equality: 

SemiDec : V {I} -> Indexed I -» Set 
SemiDec r = V i — > (x y : r i) — » Maybe (x = y) 
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That Ls, for all possible Indices, given two Indexed functors we either return a proof of 
their equality or fall. 

The type constructor Is a variation on _||_ Section 6.1.4 adapted to the type of 
SemiDec, necessary for the Fix alternative: 

infixr 5 _!/(_ 

Jf_ : {I J : Set} {r : Indexed I} {s : Indexed J} -» 

SemiDec r — » SemiDec s — > SemiDec (r | s) 
(fjfg)(inji i) = f i 
(f ]/[ g) (inj 2 i) = g i 

Decldable equality Is Impossible on Z, trivial on U, and dispatches to the supplied 
function f on the selected Index i for I: 

deqt Z f o () y 

deqt U f o tt tt = just refl 

deqt (I i) f o x y = f i x y 

Re-lndexlng proceeds recursively on the arguments, after adjusting the recursive equality 
and changing the output Index: 

deqt (F/f\g)hoxy = deqt F (h o f) (g o) x y 

Sums are only equal when both components have the same constructor. In that case, we 
recursively compute the equality, and apply congruence to the resulting proof with the 
respective constructor: 

deqt (F © G) f o (in^ x) (inj 2 y) = nothing 

deqt (F © G) f o (inj 2 x) (in^ y) = nothing 

deqt (F © G) f o (inji x) (inji y) = deqt F f o x y »= just o cong inji 

deqt (F © G) f o (inj 2 x) (inj 2 y) = deqt G f o x y »= just o cong inj 2 

We use _»=_ : {A B : Set} -> Maybe A -» (A -> Maybe B) -> Maybe B as the 
monadic bind to combine Maybe operations. The product case follows similarly, using a 
congruence lifted to two arguments: 

deqt (F <g> G) f o (x-i , x 2 ) (y-| , y 2 ) = deqt F f o xi yi 

A I -> deqt G f o x 2 y 2 »= 
A r — > just (cong 2 _,_ I r) 

A composition F © G represents a code F containing Gs at the recursive positions. 
Equality on this composition Is the equality on F using the equality on G for the recursive 
positions. Tagging Is trivial after pattern-matching: 

deqt (F © G) f o x y = deqt F (deqt G f) o x y 
deqt(!o) f .o refl refl = just refl 
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Note the .0 pattern, meaning that the second 0 is necessarily the same as the first one, 
due to equality constraints introduced by the pattern-match on refl. 

For E, we first check if the witnesses (the first components of the dependent pair) are 
equal. We make essential use of the fact that the type of the witnesses is representable 
within the universe, so we can reuse the decidable equality function we are just defining. 
If the witnesses are equal, we proceed to compare the elements, using the code produced 
by the index, and apply congruence with some to the resulting proof: 

deqt (E { C = C}g)fo (some { h } x) (some { \ 2 } y) 
with deqt {r = A _ — > _} C (A ()) tt h i 2 

deqt (L g) f 0 (some {i} x) (some y) | nothing = nothing 
deqt (L g) f 0 (some { i } x) (some y) | just refl = deqt (g i) f 0 x y 

»= just o cong some 

Equality for fixed points uses the auxiliary operator _ft_ introduced before to apply f to 
parameters and deqt to recursive calls: 

deqt (Fix F) f 0 ( x } ( y } = deqt F (f ft deqt (Fix F) f) 0 x y »= just o cong (_) 

Finally, equality for Iso uses the proof of equivalence of the isomorphism. We omit that 
case here as it is lengthy and unsurprising. 

We are now ready to test our decidable equality function. We start by instantiating 
it to natural numbers: 

deqtN : (m n : I J) -> Maybe (m = n) 
deqtN = deqt {r = (A ())} 'N' (A ()) tt 

The type of natural numbers has no input indices, therefore we supply the absurd function 
A () as the argument for the equality on the inputs. The function works as expected: 

deqtExamplei : deqtN (su ze) (su ze) = just refl 
deqtExamplei = refl 

deqtExample2 : deqtN (su ze) ze = nothing 
deqtExample2 = refl 

For lists of naturals, we supply the deqtN function we have just defined as the argument 
for the equality on input indices: 

deqtList : {A : Set} -> 

((x y : A) -> Maybe (x = y)) ->■ (h l 2 : [A]) -> Maybe (h = \ 2 ) 

deqtList {A} f x y = deqt{r = const A} 'List' (const f) tt x y 
We can now define some example lists for testing: 

h : [N] 

h = ze :: su ze :: [] 
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h ■ [N] 

I2 = ze :: ze :: [] 

And confirm that equality works as expected: 

deqtExample3 : deqtList deqtN h h = just refl 
deqtExample3 = refl 

deqtExample4 : deqtList deqtN h h = nothing 
deqtExample4 = refl 



6.4 A zipper for indexed functors 



Along with defining generic functions, we can also define type-indexed datatypes Hinze 



et a I. 2002: types defined generically in the universe of representations. A frequent 
example is the type of one-hole contexts, described for regular types by McBride |2001 



Huet 1997 



and for multireel by Rodriguez Yakushev et a I. |2009 , with application to the zip 



per 

We revisit the zipper in the context of the indexed universe, starting with the type- 
indexed datatype of one-hole contexts, and proceeding to the navigation functions. 



6.4.1 Generic contexts 

The one-hole context of an indexed functor is the type of values where exactly one input 
position is replaced by a hole. The idea is that we can then split an indexed functor 
into an input value and its context. Later, we can identify a position in a datatype by 
keeping a stack of one-hole contexts that gives us a path from the subtree in focus up 
to the root of the entire structure. 

We define Ctx as another interpretation function for our universe: it takes a code and 
the input index indicating what kind of position we want to replace by a hole, and it 
returns an indexed functor: 



Ctx : { I 0 : 


Set} - 


Ctx Z i r 0 


= _L 


Ctx U i r 0 


= _L 


Ctx (! 0') i ro 


= _L 


Ctx I i') i ro 





I ► o 



I 



I > o 



For the void, unit, and tag types there are no possible holes, so the context is the empty 
datatype. For I, we have a hole if and only if the index for the hole matches the index 
we recurse on. If there is a hole, we want the context to be isomorphic to the unit type, 
otherwise it should be isomorphic to the empty type. An equality type of i and i' has the 
desired property. 

For a re-indexed code we store proofs of the existence of the new indices together 
with the reindexed context: 
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Ctx (F T f V g) i r o = 3 (A i' -» 3 (A o' 



f i' 



i x g o = o' x Ctx F i' (r o f) o')) 



As described by McBride 2001 , computing the one-hole context of a polynomial 
functor corresponds to computing the formal derivative. This correspondence motivates 
the definitions for sum, product, and composition. Notably, the context for a composition 
follows the chain rule: 



Ctx (F © G) i r o = Ctx F i r o W Ctx Giro 

Ctx (F ® G) i r o = Ctx Firox [G] ro W [F] ro x Ctx Giro 

Ctx (F © G) i r o = 3 (Am — > Ctx F m ([ G ] r) o x Ctx G i r m) 

The context of a E f is the context of the resulting code f i', for some appropriate 
index i'. The context of an Iso is the context of the inner code: 

Ctx (E f) i r o = 3 (A i' Ctx (f i') i r o) 

Ctx (Iso C D e) i r o = Ctx C i r o 

The context of a fixed point is more intricate. Previous zippers (such as that of 
multirec) have not directly considered the context for a fixed point, since these were 
outside the universe. If we are interested in positions of an index i within a structure 
that is a fixed point, we must keep in mind that there can be many such positions, and 
they can be located deep down in the recursive structure. A Fix F is a layered tree of 
F structures. When we finally find an i, we must therefore be able to give an F-context 
for i for the layer in which the input actually occurs. Now F actually has more inputs 
than Fix F, so the original index i corresponds to the index inj-j i for F. We then need 
a path from the layer where the hole is back to the top. To store this path, we define 
a datatype of context-stacks Ctxs. This stack consists of yet more F-contexts, but each 
of the holes in these F-contexts must correspond to a recursive occurrence, i.e. an index 
marked by inj2: 

Ctx (Fix F) i r o = 3 (A j -> Ctx F (in}, i) (r | \i F r) j x Ctxs F j r o) 

data Ctxs {I O : Set} (F : (I W O) ► O) (i : O) (r : Indexed I) : Indexed O where 
empty : Ctxs F i r i 

push : {j o : 0} -> Ctx F (inj 2 i) (r | /j F r) j -> Ctxs F j r o — > Ctxs F i r o 

Note that the stack of contexts Ctxs keeps track of two output indices, just like a single 
context Ctx. A value of type Ctxs F i r o denotes a stack of contexts for a code F, focused 
on a hole with type index i, on an expression of type index o. This stack is later reused 
in the higher-level navigation functions {Section 6.4.4| . 



6.4.2 Plugging holes in contexts 

A basic operation on contexts is to replace the hole by a value of the correct type; this 
is called "plugging". Its type is unsurprising: given a code C, a context on C with hole 
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of type index i, and a value of this same index, plug returns the plugged value as an 
interpretation of C: 

plug : {I O : Set} {r : Indexed 1} {i : 1} {o : 0} ->• 
(C : I ► Oj — » Ctx C i r o — > r i — > [ C ] r o 

Plugging is not defined for the codes with an empty context type. For I, pattern- 
matching on the context gives us a proof that the value to plug has the right type, so 
we return it: 

plug Z () r 
plugU () r 
plug (! o) () r 
plug (I i) refl r = r 

Re-indexing proceeds plugging recursively, after matching the eguality proofs: 

plug (F / f \, g) (some (some (refl , refl , c))) r = plug F c r 

Plugging on a sum proceeds recursively on the alternatives. Plugging on a product 
has two alternatives, depending on whether the hole lies on the first or on the second 
component. Plugging on a composition F © G proceeds as follows: we obtain two 
contexts, an F-context c with a G-shaped hole, and a G-context d with a hole of the type 
that we want to plug in. So we plug r into d, and the resulting G is then plugged into c: 

plug (F © G) (inj-i c) r = in^ (plug F c r) 

plug (F © G) (inj 2 c) r = inj 2 (plug G c r) 

plug (F <g> G) (inji (c , g)) r = plug F c r , g 

plug (F <g> G) (inj 2 (f , cj) r = f , plug G c r 

plug (F ® G) (some (c , d)) r = plug F c (plug G d r) 

Plugging into a fixed-point structure is somewhat similar to the case of composition, 
only that instead of two layers, we now deal with an arbitrary number of layers given 
by the stack. We plug our r into the first context c, and then unwind the stack using an 
auxiliary function unw. Once the stack is empty we are at the top and done. Otherwise, 
we proceed recursively upwards, plugging each level as we go: 

plug {r = s}{o = o} (Fix F) (some { m } (c , cs)) r = unw m cs ( plug For) 
where unw : Vm ^ Ctxs Fmso — > [[ Fix F ] s m — > J Fix F ] s o 
unw .o empty x = x 

unw m (push {o} c cs) x = unw o cs ( plug F c x } 

Finally, plugging on E proceeds recursively, using the code associated with the index 
packed in the context. For isomorphisms we proceed recursively on the new code and 
apply the to conversion function to the resulting value: 

plug (L f) (some { i } c) r = some (plug (f i) c r) 
plug {r = s}{o = o} (IsoCDe)xr with e s o 
. . . | ep = to ep (plug C x r) where open _~_ 
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6.4.3 Primitive navigation functions: first and next 

With plug we can basically move up in the zipper: after plugging a hole we are left with 
a value of the parent type. To move down, we need to be able to split a value Into Its 
first child and the rest. This Is the task of first: 

first : {A I O : Set} {r : Indexed 1} {o : 0} -> (C : I ► O) 

((i : I) -> r i -> Ctx C i r o -> Maybe A) -> [ C ] r o -» Maybe A 

We write first In continuation-passing style. One should read It as a function taking a 
value and returning a context with a hole at the first (i.e. leftmost) possible position, the 
value previously at that position, and Its Index. These are the three arguments to the 
continuation function. Since not all values have children, we might not be able to return 
a new context, so we wrap the result In a Maybe. Note that first (and not the caller) 
picks the Index of the hole according to the first input that It can find. 

As there are no values of void type, this case Is Impossible. For unit and tag types, 
there are no elements, so the split falls: 

first Z k () 

first U kx = nothing 

first (! o) k x = nothing 

For I there Is exactly one child, which we return by Invoking the continuation: 

first (I i) k x = k i x refl 

For re-lndexlng and sums we proceed recursively, after adapting the continuation 
function to the new Indices and context appropriately: 

first (F / f V g) k x = first F (A i' r c -* k (f i') r (some (some (refl , (refl , c))))) x 
first (F © G) k (in^ x) = first F (A i r c -> k i r (in^ c)) x 
first (F © G) k (inj 2 x) = first G (A i r c -> k i r (inj 2 c)) x 

On a product we have a choice. We first try the first component, and only In case of 
failure (through plusMaybe) we try the second: 

first (F ® G) k (x , y) = plusMaybe (first F (A i r c -> k i r (inji (c , y))) x) 

(first G (A i r c — > k i r (inj 2 (x , c))) y) 

Composition follows the nested structure: we first split the outer structure, and If that 
Is successful, we call first again on the obtained Inner structure: 

first (F © G) k x = first F (A m s c — > first G (A i r d — > k i r (some (c , d))) s) x 

Fixed points regulre more care. We use two mutually-recursive functions to handle 
the possibility of having to navigate deeper Into recursive structures until we find an 
element, building a stack of contexts as we go. Note that the type of Input Indices 
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changes once we go inside a fixed point. If we obtain a split on an inji -marked value, 
then that is an input index of the outer structure, so we are done. If we get a split on 
an inj2-marked value, we have hit a recursive occurrence. We then descend into that by 
calling fstFix again, and retain the current layer on the context stack: 

first {A} { I} {0} {r} {0} (Fix F) k x = fstFix x empty where 
mutual 

fstFix : {m : 0} — > ^/Frm — ► Ctxs Fmro ^ Maybe A 
fstFix ( x } cs = first F (contFix cs) x 

contFix : {m : 0} Ctxs Fmro— >(i:IWO)— > 

(r | [s F r) i — > Ctx F i (r | y F r) m — > Maybe A 
contFix cs ( i nj ^ i) r c = k i r (some (c , cs)) 
contFix cs (inj 2 i) r c = fstFix r (push c cs) 

Splitting on a Z proceeds recursively as usual, and on isomorphisms we apply con- 
version functions where necessary: 

first (L f) k (some {i'} y) = first (f i') (Aire — > k i r (some c)) y 

first {r = r} {0 = 0} (Iso C D e) k x with e r 0 
. . . | ep = first C k (from ep x) where open _~_ 

Another primitive navigation function is next, which, given a current context and an 
element which fits in the context, tries to move the context to the next element to the 
right, producing a new context and an element of a compatible (and possibly different) 
type: 

next : {A I O : Set} {r : Indexed I} {0 : 0} (C : I ► O) 
((i : I) r i Ctx C i r 0 Maybe A) 
{ i : I } -» Ctx C i r 0 -» r i -> Maybe A 

Its implementation is similar to that of first, so we omit it. 



6.4.4 Derived navigation 

Given the primitives plug, first, and next, we are ready to define high-level navigation 
functions, entirely hiding the context from the user. Recursive user datatypes are gen- 
erally defined as a top-level application of Fix to another code. We thus define a zipper 
data structure that enables the user to navigate through a structure defined by a fixed 
point on the outside. We can then efficiently navigate to all the recursive positions 
in that structure. Note that variations of this approach are possible, such as a zipper 
that also allows navigating to parameter positions, or defining navigation functions that 
operate on an Iso code. 

While we are traversing a structure, we keep the current state in a datatype that 
we call a location. It contains the subtree that is currently in focus, and a path up to 
the root of the complete tree. The path is a stack of one-hole contexts, and we reuse 
the Ctxs type from Section 6.4.1 to hold the stack: 
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6.4 A zipper for indexed functors 



data Loc {I O : Set} (F : (I W O) ► O) (r : Indexed I) (o : O) : Set where 

loc : {o' : 0} — > [ Fix F ] r o' -» Ctxs F o' r o —> Loc F r o 

The high-level navigation functions all have the same type: 
Nav : Seti 

Nav = V {I 0} {F : (I W O) ► 0} {r : Indexed I } { o : 0} — » 
Loc F r o -* Maybe (Loc F r o) 

Given a location, we might be able to move to a new location, keeping the same code, 
interpretation for recursive positions, and output type index. We need to allow for failure 
since, for instance, it is not possible to move down when there are no children. 
Moving down corresponds to splitting the context using first: 

down : Nav 

down {F = F} (loc ( x } cs) = first F (A {(inji i) r d -* nothing 

; (inj 2 i) r d -* just (loc r (push d cs))}) x 

Here we can use Agda's syntax for pattern matching on anonymous lambdas when 
defining the continuation function. This function expresses the behavior for the different 
kinds of input positions: we do not descend into parameters, and on recursive calls we 
build a new location by returning the tree in focus and pushing the new layer on the 
stack. 

Moving up corresponds to plugging in the current context. It fails if there is no context, 
meaning we are already at the root: 

up : Nav 

up (loc x empty) = nothing 

up { F = F} (loc x (push c cs)) = just (loc (( plug F c x }) cs) 

Moving to the right corresponds to getting the next child. We process the results 
of next as in down, this time using an auxiliary definition for the continuation: 

right : Nav 

right (loc x empty) = nothing 

right {I} {0} {F} {r} {o} (loc x (push {m} c cs)) = next F auxf c x 
where auxf : (i : I W O) — > (r | jj F r) i — > 

Ctx F i (r | // F r) m — > Maybe (Loc F r o) 
auxf (inj! i) r d = nothing 
auxf (inj2 i) r d = just (loc r (push d cs)) 

The functions presented so far allow reaching every position in the datatype. Other 
navigation functions, such as to move left, can be added in a similar way. 
Finally, we provide operations to start and stop navigating a structure: 

enter : V {I 0} {F : (I W O) ► 0} {r : Indexed I} {o : 0} -> 
[ Fix F ] r o —* Loc F r o 
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6 Indexed functors 

enter x = loc x empty 

leave : V {I 0} {F : (I W O) ► 0} {r : Indexed 1} {o : 0} -» 

Loc F r o -» Maybe ([ Fix F ] r o) 
leave (loc x empty) = just x 

leave (loc x (push h t)) = up (loc x (push h t)) »= leave 

To enter we create a location with an empty context, and to leave we move up until the 
context Is empty. 

It Is also useful to be able to manipulate the focus of the zipper. The function update 
applies a type-preserving function to the current focus: 

update : V {I 0} {F : (I W O) ► 0} {r : Indexed 1} -> 

[ Fix F ] r =4 I Fix F ] r -> Loc F r =4 Loc F r 
update f _ (loc x I) = loc (f _ x) I 



6.4.5 Examples 

We now show how to use the zipper on the Rose datatype of Section 6.1 .6 The rep- 



resentation type of rose trees Is non-trlvlal since It uses composition with lists (and 
therefore contains an Internal fixed point). Let us define an example tree: 



treeB : 
treeB t 



tree 
tree 



Rose N 
= fork 5 

(fork 4 

Rose N 

= treeB (fork 0 [ 



Rose N 



(fork 3 [] :: (fork 2 [] :: (fork 1 



(t " 0) " [])))) 



Our example tree has a node 5 with children numbered 4 through 1. The last child has 
one child of Its own, labelled 0. 

We now define a function that navigates through this tree by entering, going down 
(Into the child labelled 4), moving right three times (to get to the rightmost child), 
descending down once more (to reach the child labelled 0), and finally Increments this 
label: 



navTree : Rose N — > Maybe (Rose N) 

navTree t = down (enter (fromRose t)) 

»= right »= right »= right »= down 

»= just o update (const (map 'Rose' (const su) tt)) _ 

»= leave »= just o toRose 

Since the navigation functions return Maybe, but other functions (e.g. enter and fromRose) 
do not, combining these functions regulres care. However, It would be easy to define 
a small comblnator language to overcome this problem and simplify writing traversals 
with the zipper. 
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6.5 Conclusion and future work 



We can check that our traversal behaves as expected: 

navTreeExample : navTree tree = just (treeB (fork 1 [ 
navTreeExample = refl 



We can also use the zipper on the Perfect nested datatype of Section 6.1 .9 for instance 



In this example, we enter a perfect tree, move down into the first (thus leftmost) child, 
swap all the elements of this subtree, and then leave: 

swapLeft : {n : N} ->■ Perfect N {n} ->• Maybe (Perfect N {n}) 
swapLeft p = down (enter (fromPerfect p)) 

»= just o (update f _) 

»= leave 

»= just o toPerfect 

where f : (n : N) -> ['Perfect' ] (T N) n ->■ ['Perfect' J (T N) n 

f n p = fromPerfect (cataPerfect {R = A n — > Perfect N {n}} leaf 

(A a —> split a o swap) (toPerfect p)) 

For swapping we use a catamorphism with the function cataPerfect, which is just a 
specialisation of cata to the type of perfect trees. We can check that swapLeft behaves 
as expected: 

p 0 : Perfect N 

Po = split 0 (leaf , leaf) 

Pi : Perfect N 

Pi = split 1 (leaf , leaf) 

p : Perfect N -> Perfect N -> Perfect N 

pxy = split 6 (split 2 (x , y) , split 5 (po , Pi )) 

swapLeftExample : swapLeft (p p 0 Pi) = just (p pi p 0 ) 
swapLeftExample = refl 

We have seen how to define a zipper for the universe of indexed. In particular, we 
have seen a type of one-hole contexts for fixed points, using a stack of contexts that is 
normally used only for the higher-level navigation functions. Even though this zipper is 
more complex than that of multirecQit operates through all the codes in this universe, 
meaning that we can now zip through indexed datatypes, for instance. 



6.5 Conclusion and future work 

In this chapter we have seen a universe of indexed functors for dependently typed generic 
programming in Agda which is both intuitive, in the sense that its codes map naturally to 
datatype features, and powerful, since it supports a wide range of datatypes and allows 
defining a wide range of datatype-generic behavior. 

1 http : //hackage . haskell . org/package/zipper 
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6 Indexed functors 



The key features of the indexed approach are: support for parametrised datatypes 
and recursive positions in a uniform way, fixed points as part of the universe, support for 
general composition, and the possibility to incorporate isomorphisms between datatypes 
into the universe. These features make it possible to reuse codes once defined, and to 
make normal user-defined Agda datatypes available for use in generic programs. 

Along the way we have seen how proofs of correctness easily integrate with indexed 
functors, both in the universe and in the generic functions defined. Furthermore, we 
have shown a generalisation of the zipper operation to the indexed universe, allowing 
for efficient and type-safe generic traversals. 



Options for future work include generalising from the zipper to dissection McBride 



2008 as well as refining the universe further, for example by allowing generalisation 
over arity and datatype kind, both present in the work of Weirich and Casinghino |201 0 



Furthermore, we lack a mechanism for automatic generation of datatype representations, 
and we cannot precisely state which datatypes are not representable in our universe. 

The original goal that inspired the indexed approach was to overcome the limitations 
of multirec, which can handle mutually recursive families of datatypes, but does not 
support parametrised datatypes or composition. While indexed overcomes these limi- 
tations, it is only implemented in Agda. Agda clearly has many advantages for generic 
programming, but Haskell is currently still superior when it comes to writing practical 
code. We hope that the newly introduced kind-level features in GHC 7.4 jYorgey et al. 



201 2| will allow us to obtain a reasonably elegant encoding of indexed in Haskell, 



bringing the power of this approach to a wider audience. 
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CHAPTER 7 



The instant-generics library 



instant-generics is another approach to generic programming with type families, 
initially described by Chakravarty et a I. |2009 . It distinguishes itself from the other 
approaches we have discussed so far in that it does not represent recursion via a fixed- 
point combinator. Like regular, instant-generics has also been used to implement a 
generic rewriting library Van Noort et a I. 201 0 . To allow meta-variables in rewrite rules 
to occur at any position (i.e. not only in recursive positions), type-safe runtime casts are 
performed to determine if the type of the meta-variable matches that of the expression. 
The generic programming support recently built into the Glasgow and Utrecht Haskell 
compilers, which we discuss in Chapter TT| is partly inspired by instant-generics, 
and shares a number of features. 



7.1 Agda model 



In the original encoding of instant-generics by |Chakravarty et al. in Haskell, recur- 
sive datatypes are handled through indirect recursion between the conversion functions 
(from and to) and the generic functions. This is a form of a shallow encoding, but dif- 
ferent from the one we have seen in Section 3.2 in regular we know the type of the 
recursive elements, so we carry around a function to apply at the recursive positions, 
instant-generics, on the other hand, relies on Haskell's class system to recursively 
apply a generic function at the recursive points of a datatype. 



We find that the most natural way to model this in Agda is to use coinduction Daniels 



son and Altenkirch 2010. This allows us to define infinite codes, and generic functions 



operating on these codes, while still passing the termination check. This encoding would 
also be appropriate for other Haskell approaches without a fixed-point operator, such as 



75 



7 The instant-generics library 



Generics for the Masses" Hinze 2006 and "Lightweight Implementation of Generics 



and Dynamics" |Cheney and Hinze 2002. Although approaches without a fixed-point 
operator have trouble expressing recursive morphisms, they have been popular in Haskell 
because they easily allow encoding datatypes with irregular forms of recursion (such as 
mutually recursive or nested Bird and Meertens 1998 datatypes). 
Coinduction in Agda is built around the following three operations: 



oo : V ( A : Set) -> Set 

tt : V {A : Set} — > A — > oo A 

b : V {A : Set} — > oo A — > A 



The oo operator is used to label coinductive occurrences. A type oo A can be seen as a 
delayed computation of type A, with associated delay (ft) and force (b) functions. Since 
Agda is a total language, all computations are reguired to terminate; in particular, values 
are reguired to be finite. Coinduction lifts this restriction by allowing the definition of 
coinductive types which need not be finite, only productive. Productivity is checked 
by the termination checker, which reguires corecursive definitions to be guarded by 
coinductive constructors. The universe of codes for instant-generics has a code R for 
tagging coinductive occurrences of codes: 



data Code : Seti where 



u 






Code 


K 


: Set 




-» Code 


R 


: (C : 


oo Code) 


— > Code 


_®_ 


: (C D 


: Code) 


— > Code 


_®_ 


: (C D 


: Code) 


— > Code 



Compared to the previous approaches, the code K for arbitrary Sets is also a novelty. 
We have not introduced a code for constants in any of the Agda models so far because 
its inclusion is trivial and unsurprising. For instant-generics, however, it becomes 
necessary for the embeddings in |Chapter 8| 

We give the interpretation as a datatype to ensure that it is inductive. The judicious 
use of the coinduction primitive b makes the Agda encoding pass the termination checker, 
as the definitions remain productive: 



data [_] : Code — > Seti where 



tt 








[u 
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{A : 


Set} -» 


A 


- [KA 




rec 


{C : 


oo Code} — > 


[bC] 


-» IRC 




inji 


{C D 


: Code} ->• 


[c] 


- [c® 
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inj 2 


{C D 


: Code} -> 


[D] 


ic© 


D 




{C D 


: Code} -> 


IC] - 


[D] - [CO 


D 



As an example, consider the encoding of lists in instant-generics: 
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7.1 Agda model 



'List' : Set -> Code 

'List' A = U © (K A ® R (J 'List' A)) 

Although the definition of 'List' Is directly recursive, It Is accepted by the termination 
checker, since It remains productive. 

Due to the lack of fixed points, we cannot write a map function on the recursive 
positions. But we can easily write other recursive generic functions, such as a traversal 
that crushes a term Into a result of a fixed return type: 

crush : {R : Set} 

(C : Code) -^(R^R^R)^(R^R)^R^[C]]^R 

crush U _ffl_ ft 1 _ =1 
crush (K _) _ffl__ ft 1 _ =1 

crush (R C) _ffl_ ft 1 (rec x) = ft (crush (b C) _ffl_ ft 1 x) 
crush (C © D) _ffl_ ft 1 (inj-i x) = crush C _ffl_ ft 1 x 
crush (C © D) _ffl_ ft 1 (inj 2 x) = crush D _ffl_ ft 1 x 

crush (C ® D) _ffl_ ft 1 (x , y) = (crush C _ffl_ ft 1 x) ffl (crush D _ffl_ ft 1 y) 

Function crush Is similar to map In the sense that It can be used to define many generic 
functions. It takes three arguments that specify how to combine pairs of results (_EH_), 
how to adapt the result of a recursive call (ft), and what to return for constants and 
constructors with no arguments (!)[]] However, crush Is unable to change the type of 
datatype parameters, since instant-generics has no knowledge of parameters. 
We can compute the size of a structure as a crush, for Instance: 

size : (C : Code) -> [ C ] -> N 
size C = crush C _+_ su ze 

Here we combine multiple results by adding them, Increment the total at every recursive 
call, and Ignore constants and units for size purposes. We can test that this function 
behaves as expected on lists: 

aList : [ 'List' T ] 

aList = inj 2 (k tt , rec (inj 2 (k tt , rec (inji tt)))) 

testSize : size _ aList = 2 
tests ize = refl 

While a map function cannot be defined like In the previous approaches, traversal and 
transformation functions can still be expressed In instant-generics. In particular, if 
we are willing to exchange static by dynamic type checking, type-safe runtime casts can 
be performed to compare the types of elements being mapped against the type expected 
by the mapping function, resulting In convenient to use generic functions |Van Noort 



et al. 2010. However, runtime casting Is known to result In poor runtime performance, 



as It prevents the compiler from performing type-directed optimisations (as those of 



Chapter 9| 



1 lt is worth noting that crush is itself a simplified catamorphism for the Code type. 
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7 The instant-generics library 



7.2 Haskell implementation 

The instant-generics library is encoded in Haskell with the following representation 
types: 



data U 
data Var a 
data Rec a 
data a :+: 0 
data a : x : /? 



= U 

= Var a 

= Rec a 

= L a | R jS 

= a :x: /3 



The main difference from the previous approaches is that we no longer need to carry 
additional parameters to encode recursion. Every argument to a constructor is wrapped 
in a Var or Rec tag to indicate if the argument is a parameter of the type or a (potentially 
recursive) occurrence of a datatype. These correspond to the codes K and R in our Agda 
model, respectively. 



7.2.1 Datatype representation 

As customary, we use a type class to mediate between a value and its generic repre- 
sentation: 

class Representable a where 
type Rep a 

to :: Rep a — » a 
from :: a — » Rep a 

A Representable type has an associated Representation type, which is constructed using 
the types shown before. We use a type family to encode the isomorphism between a 
type and its representation, together with conversion functions to and from. 
As an example, we show the instantiation of the standard list datatype: 

instance Representable [ a } where 
type Rep [a] = U :+: (Var a : x: Rec [ a }) 

from [] = L U 

from (h : t) = R (Var h :x: Rec t) 

to (L U) = [] 

to (R (Varh :x: Rec t)) = h : t 

Lists have two constructors: the empty case corresponds to a Left injection, and the cons 
case to a Right injection. For the cons case we have two parameters, encoded using a 
product. The first one is a Variable, and the second a Recursive occurrence of the list 
type. 
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7.2 Haskell Implementation 



7.2.2 Generic functions 

Generic functions are defined by giving an Instance for each representation type. To 
define the crush function we have seen In the Agda model we first define a type class: 

class CrushRep a where 

crushRep :: (p — > p — > p) — > (p — > p) — > p — > cr — > p 

We call this class CrushRep because we only give Instances for the representation types. 
The Instances for units, sums, and products are unsurprising: 

instance CrushRep U where 
crushRep z _ = z 

instance (CrushRep a, CrushRep /?) => CrushRep (a :+: /?) where 
crushRep f g z (L x) = crushRep f g z x 
crushRep f g z (R x) = crushRep f g z x 

instance (CrushRep a, CrushRep ft) => CrushRep (a :x: /?) where 
crushRep f g z (x :x: y) = f (crushRep fgzx) (crushRep f g z y) 

For variables we also stop the traversal and return the unit result: 

instance CrushRep (Var a) where 
crushRep f g z (Var x) = z 

For recursive types, however, we have to recursively apply crushRep. In the previ- 
ous approaches we carried around a function to apply at the recursive Indices. In 
instant-generics we rely on Indirect recursion: 

instance (Crush a) => CrushRep (Rec a) where 
crushRep f g z (Rec x) = g (crush fgzx) 

We use another class, Crush, to define how to crush user datatypes. Since the repre- 
sentation In instant-generics Is shallow, at the Rec positions we have user types, 
not representation types. So we recursively Invoke the crush function, which performs 
the same task as crushRep, but on user datatypes. 
The user-facing Crush class Is defined as follows: 

class Crush a where 

crush :: (p — > p — > p) — > (p — > p) — > p — > a — > p 

We also define a function crushDefault that Is suitable for using when defining generic 
Instances of Crush: 

crushDefault :: (Representable a, CrushRep (Rep a)) => 

(P P ->■ P) (P P) P ->■ a -> p 
crushDefault f g z = crushRep f g z o from 
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7 The instant-generics library 

Instantiating this generic function to representable datatypes is now very simple: 

instance Crush [ a ] where 
crush = crushDefault 

To finish the same example as in the Agda model, we define a function to compute the 
size of a term as a crush: 

size :: (Crush a) => a — > Int 
size = crush (+) (+1) 0 

Now we can confirm that size (1:2: []) evaluates to 2. 

We have seen a library for generic programming that does not use a fixed-point view 
on data. This gives us more flexibility when encoding datatypes, but we lose the ability 
to express certain generic behaviour, such as recursive morphisms. 
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CHAPTER 8 



Comparing the approaches 



We have seen five different libraries for generic programming: regular {Chapter^) , 
polyp (Chapter 4) , multirec {Chapter"?) , indexed {Chapter!)) , and instant-generics 
{Chapter"^ We have modelled each of these libraries in Agda by reducing the ap- 
proach to its core elements (universe and its interpretation). We have also seen example 
datatype encodings and generic functions. In this chapter we relate the approaches to 
each other, formally, by providing conversion functions between approaches and proofs 
of correctness. 



8.1 Introduction 



The abundance of generic programming approaches is not a new problem. Including 
pre-processors, template-based approaches, language extensions, and libraries, there 
are well over 15 different approaches to generic programming in Haskell Brown and 



Sampsonj |2009]|Chakravarty et al.||20"09"l|Cheney and Hinze||2002]|Hinze||2006||Hinze 



and Loh 2006 Hinze and Peyton Jones 2001 Hinze et a I. 2006 ansson and Jeuring 



1997 Kiselyov 


2006 


Lammel and Peyton Jones 2003 2004 


2005 Loh 2004 


Maga- 


lhaes et a I. 


20' 


OalllV 


itchell and Runciman 


2007| INorell and 


Janssonl |2004| |Oliveira 


et al. 2006 


jRodriguez Yakushev et a I. 2009 


Weirich 2006. This abundance is 


caused 



by the lack of a clearly superior approach; each approach has its strengths and weak- 
nesses, uses different implementation mechanisms, a different generic view |Holdermans 
et a I. 2006 (i.e. a different structural representation of datatypes), or focuses on solving 



a particular task. Their number and variety makes comparisons difficult, and can make 
prospective generic programming users struggle even before actually writing a generic 
program, since first they have to choose a library that is adeguate to their needs. 
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8 Comparing the approaches 



Some effort has been made in comparing different approaches to generic programming 

or 



from a practical point of view Hinze et al. 2007 Rodriguez Yakushev et a I. 



to classify approaches Hinze and Loh 2009. While Generic Haskell Loh 



2008 



2004 has 



been formalised in different attempts Verbruggen et a I. 2010 Weirich and Casinghino 



2010, no formal comparison between modern approaches has been attempted, leaving a 



gap in the knowledge of the relationships between each approach. We argue that this 
gap should be filled; for starters, a formal comparison provides a theoretical foundation 
for understanding different generic programming approaches and their differences and 
similarities. However, the contribution is not merely of theoretical interest, since a 
comparison can also provide means for converting between approaches. Ironically, code 
duplication across generic programming libraries is evident: the same function can be 
nearly identical in different approaches, yet impossible to reuse, due to the underlying 
differences in representation. With a formal proof of inclusion between two approaches 
a conversion function can be derived, removing the duplication of generic code, as we 
show in ISection 8.3I 

In this chapter we take the initial steps towards a formal comparison of generic pro- 
gramming approaches. We establish relationships among the five Agda encodings shown 
before, and reason about them. While the inclusion relations are the ones we expect, the 
way to convert between approaches is often far from straightforward, and reveals subtle 
differences between the approaches. Each inclusion is evidenced by a conversion func- 
tion that brings codes from one universe into another, enabling generic function reuse 
across different approaches. 

Our proofs are fully machine-checked (in Agda), but written in eguational reasoning 
style, so that they resemble handwritten proofs, and remain clear and elegant. A few 
caveats remain, namely with regard to termination and universe levels inconsistency, 
which we discuss in more detail in ISection 8.4I 



8.2 Embedding relation 

In this section we show which approaches can be embedded in other approaches. When 
we say that approach A embeds into approach B, we mean that the interpretation of any 
code defined in approach A has an eguivalent interpretation in approach B. The starting 
point of an embedding is a code-conversion function that maps codes from approach A 
into approach B. 

For this comparison we consider the Agda models for the approaches presented before. 
The indexed approach has been shown in full detail, while the other four approaches 
have been stripped down to their core. Therefore we consider only a subset of indexed 
for this section. The minimised indexed universe we use in this section follows: 



data Codei (I O : Set) 
U 

I 

: O 



Seti where 
Codei 
-> Code 
— > Code 



(F G : Code, I O) -» Code 



I O 
I O 
I O 
I O 
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8.2 Embedding relation 



Fix 



(F G : Codei I O) ->• Code: I O 

{M : Set} ->• (F : Codei MO)^(G 

(F : Codei (I WO)0) -> Code; I O 



Code: I M) -» Code: I O 



Recall that we used an infix type operator _►_ to represent the universe of indexed; for 
consistency, we name that operator Codei In this section. We elide Z as this code does 
not add much expressiveness to the universe, and is not present in the other approaches. 
Isomorphisms (Iso) are also removed for simplicity, as they do not add expressive power, 
only convenience. The codes for Z and reindexing (_/'_\_), however, add considerable 
power to the indexed universe. Yet we decide to remove them for comparative purposes 
so that we can relate indexed to instant-genericsQ 

|Figure 8~ presents a graphical view of the embedding relation between the five ap- 
proaches; the arrows mean "embeds into". Note that the embedding relation is naturally 
transitive. As expected, multirec and polyp both subsume regular, but they don't 
subsume each other, since one supports families of recursive types and the other sup- 
ports one parameter. They are however both subsumed by indexed. Finally, the liberal 
encoding of instant-generics allows encoding at least all the types supported by the 
other approaches (even if it doesn't support the same operations on those types, such 
as catamorphisms). 






indexed 

In Pn ® ® ® Fix 




instant-generics 

R © ® 


> 



Figure 8.1: Embedding relation between the approaches 
We now discuss each of the five conversions and their proofs. 



1 The minimised indexed shown in this section identifies a subset of full indexed that is expressive enough 
to generalise regular, polyp, and multirec, while remaining embedabble into instant-generics. 



83 



8 Comparing the approaches 



8.2.1 Regular to PolyP 

We start with the simplest relation: embedding regular into polyp. The first step is 
to convert regular codes into polyp codes: 

flV-.p : Code r -* Code p 

ftr^pUr = Up 

iTr— >p 'r = 'p 

ftr^p(F© r G) = (t M pF)©p(t^G) 
fU P (F® r G) = (t,.pF)« p (tr.pG) 

Since all libraries share similar names, we use subscripts to denote the library we are 
referring to; r for regular and p for polyp. All regular codes embed trivially into 
polyp, so ftr^p is unsurprising. After having defined the code conversion we can show 
that the interpretation of a code in regular is eguivalent to the interpretation of the 
converted code in polyp. We do this by defining an isomorphism (see Section 2.2.3) 



between the two interpretations (and their fixed points). We show one direction of the 
conversion, from regular to polyp: 



from r ^p : {a : Set} (C : Code r ) -> [Cj r a I f^p C] p 1 a 

from r ^p U r tt = tt 

from r _^ p l r x = x 

fron-i^p (F© r G) (inj-i x) = i nj-i (from r ^ p F x) 

from^p (F© r G) (inj 2 x) = inj 2 (from r ^ p G x) 

frorPr-.p (F <S>, G) (x , y) = from r _> p F x , from r ^ p G y 

from/y r ^p : (C : Code r ) -> ;/ r C (ft r ^ p C) _L 

from^ r ^p C ( x } r = ( from r _, p C (map r C (fromp r ^ p C) x) } p 

Since regular does not support parameters, we set the polyp parameter to _L for 
regular codes. Function from r _> p does the conversion of one layer, while from/y r _> p ties 
the recursive knot by expanding fixed points, converting one layer, and mapping itself 
recursively. Unfortunately from/j r ^ p (and indeed all conversions in this section involving 
fixed points) does not pass Agda's termination checker; we provide some insights on how 
to address this problem in |Section 8.4| 

The conversion in the other direction (to r _» p and to/J r _» p ) is entirely symmetrical: 

to r ^p : {a : Set} (C : Code r ) -> I fo^ p Cj p _L a -> [Cj r a 

tO r ^p U r tt = tt 

tO r ^p l r X = X 

to r ^p (F ® r G) (inji x) = inj! (to r ^ p F x) 
to r ^ p (F © r G) (inj 2 x) = inj 2 (to r ^ p G x) 
to r ^ p (F ® r G) (x , y) = to r ^ p F x , to r ^ p G y 

to// r _ p : (C : Code r ) ;/ p (ft r _> p C) _L -» \i, C 

to/y r ^p C ( x } p = ( to r ^p C (mapp (flv^p C) id (to/y r ^ p C) x) } r 
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Having defined the conversion functions, we now have to prove that they indeed form 
an isomorphism. We first consider the case without fixed points: 

isoi r _> p : {R : Set} —> 

(C : Code r ) -> (x : [C] r R) -> to r ^ p C (from r ^ p C x) = x 

iso 1r ^ p U r _ = refl 

iso 1r ^ p l r _ = refl 

isoi r ^ p (F© r G) (inji x) = cong inji (isoi r ^ p F x) 

isoi r ^ p (F© r G) (inj 2 x) = cong inj 2 (isoi r -» P G x) 

isoi r ^p (F ® r G) (x , y) = cong 2 _,_ (isoi r ^ p F x) (isoi r _» p G y) 

This proof is trivial, and so is the proof for from r ^ p C (to r ^ p C x) = x, its counterpart for 
the other direction. 

When considering fixed points the proofs become more involved, since recursion has 
to be taken into account. However, using the eguational reasoning module from the 
standard library (see the work of Mu et a I. 2009 for a detailed account on this style of 
proofs in Agda) we can keep the proofs readable: 

open =-Reasoning 

iso^i r ^ p : (C : Code r ) (x : fj r C) — > to/j r _» p C (fromp r ^ p C x) = x 
iso/vi r ^ p C (x) r = cong (_), $ 

begin 

to r ^ p C (map p (Hv-> p C) id (top r ^ p C) (from r ^ p C (map r C (from^ p C) x))) 
=( mapCommute? C _ } 

map r C (top r ^p C) (to r ^p C (from r ^ p C (map r C (from^ r ^ p C) x))) 
=( cong (map r C (top r ^ p C)) (iso 1r ^ p C _) } 

map r C (top r ^p C) (map r C (from^ r ^ p C) x) 
=( map° C } 

map r C (top r ^p C o fromp r ^p C) x 
=( map^ C (isoA/ 1r ^p C) x } 

map r C id x 
=( mapf C } 

xl 

In this proof we start with an argument relating the maps of regular and polyp: 

mapCommute? : {a : Set} {f : a ->• £} (C : Code r ) (x : [ %^ C] p _L a) -»• 
to r ^p C (map p (ffr^p C) id f x) = map r C f (to r ^ p C x) 

In words, this theorem states that the following two operations over a regular term x 
that has been lifted into polyp are eguivalent: 
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• To map a function f over the recursive positions of x in polyp with map p and then 
convert to regular; 

• To first convert x back into regular, and then map the function f with map r . 

After this step, we either proceed by a recursive argument (referring to isoi r _^ p or 
iso/Ji r ^ p ) or by a lemma. For conciseness, we show only the types of the lemmas: 

map? : {a $ y : Set} {f : /S -> y} {g : a -> (C : Code r ) 
{x : [C] r a} — > map r C f (map r C g x) = map r C (f o g) x 

mapf : {aft : Set} {f g : a ->■ £} (C : Code r ) -> 

(V x -» f x = g x) -> (V x -* map r C f x = map r C g x) 

mapj. d : {a : Set} (C : Code r ) {x : [C] r a} -> map r C id x = x 

These lemmas are standard properties of map r , namely the functor laws and the fact 
that map r preserves extensional eguality (a form of congruence on map r ). All of them are 
easily proven by induction on the codes, similarly to the proofs for the indexed map of 
ISection 6.2I 

Put together, from/j r ^ p , to/J r ^ p , iso/Ji^p, and iso/J2 r ^p (the dual of iso/J-| r ^ p ) form an 
isomorphism that shows how to embed regular codes into polyp codes: 

Regular~PolyP : (C : Code r ) -> \i x C ~ v p (f|w p C) _L 



Regular~PolyP C = record {from 


= fromp r _ 




; to 


= to// r _ p 


c 


; isoi 


= iso^i r _ 


>P c 


; iso 2 


= iso/i 2r - 


> P c} 



8.2.2 Regular to Multirec 

The conversion from regular to multirec (subscript m) is very similar to the conversion 
into polyp. We convert a regular code into a multirec code with a unitary index type, 
since regular codes define a single type: 



1Um 


: Code r 


-> Code m T 






fWm 


U r 


= u m 






1Tr->m 


lr 


= Ltt 






ttr— >m 


(F® r G) 


= (t^ m F) € 


Dm (1tr- 


,mG) 


ftV— >m 


(F® r G) 


= flUn, F) G 


5m (TTr- 


♦ mG) 



Instead of defining conversion functions from and to, we can directly express the egual- 
ity between the interpretations: 

Tr^m : {a : Set} (C : Code r ) [Cj r a = [ ft r _> m C] m (A _ -> a) tt 
Tr^m U r = refl 

Tr^m lr = refl 
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Tr^m (F © r G) = COng 2 _b_ (Tr^m F) (Tr^m G) 
t r ^ m (F® r G) = COng 2 _X_ (tr^ m F) (tr^m G) 

We elide the isomorphism between the fixed points as it is similar to that for polyp. 
8.2.3 PolyP to Indexed 

We proceed to the conversion between polyp and indexed (subscript i) codes. As we 
mentioned before, particular care has to be taken with composition; the remaining codes 
are trivially converted, so we only show composition: 

flp^i : Code p -> Codei (TWT)T 

fTp-i (F ©p G) = (Fix (f D , F)) ®| % , G) 

We cannot simply take the indexed composition of the two converted codes because 
their types do not allow composition. A polyp code results in an open indexed code 
with one parameter and one recursive position, therefore of type Codei (T W T) T. 
Taking the fixed point of such a code gives a code of type Codei T T, so we can mimic 
polyp's interpretation of composition in our conversion, using the Fix, of indexed. In 
fact, converting polyp to indexed helps us understand the interpretation of composition 
in polyp, because the types now show us that there is no way to define a composition 
other than by combining it with the fixed-point operator. 

Converting composed values from polyp is then a recursive task, due to the presence 
of fixed points. We first convert the outer functor F, and then map the conversion onto 
the arguments, recalling that on the left we have parameter codes G, while on the right 
we have recursive occurrences of the original composition: 

from p _H : {a p : Set} (C : Code p ) -» 

[C] p a p -» I flp-M CJ, ((const a) |, (const p)) tt 

fromp^i (F@ P G) (x) p = 

( mapi (fr p ^i F) ((A _ from p ^ G) ||, (A _ -> from p _i (F © p G))) tt 
(from p ^i F x) ), 

We also show the conversion in the opposite direction, which is entirely symmetrical: 

top^j : {a p : Set} 

(C : Code p ) [ %_> t C], ((const a) |, (const p)) tt ^ IC] p ap 

top^i (F@ P G) (x)i = 

(top^i F (map, (ftp^i F) ((A _ ^ to p ^i G) ||, (A _ -> to p ^i (F ® p G))) tt x) ) p 

The conversion of polyp fixed points ignores parameters and maps itself recursively 
over recursive positions: 

from/y p ^i : {a : Set} (C : Code p ) -> y p C a -> [[ Fix, C)| (const a) tt 
from^p^i C ( x } p = ( from p ^i C (map p C id (fromp p ^i C) x) )j 
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We omit the conversion in the opposite direction for brevity, and also the isomorphism 
proof. Both the case for composition and for polyp fixed points require the properties 



of map, from Section 6.2 



8.2.4 Multirec to Indexed 

The conversion from multirec to indexed is simpler than the conversion from polyp 
into indexed because multirec does not support composition. We embed a multirec 
family indexed over a set I into an indexed family with output index I, and input index 
_L W I (no parameters, same number of outputs, before taking the fixed point): 

fU-i : {I : Set} ->• Code m I -> Code: (_L W I) I 

TTm^iU m = U| 

iTm^i Urn 0 = li (M2 i) 

ftm-nM) = hi 

lTm^i(F© m G) = (ftm-H F) ©i (Hm-H G) 

ft m -i (F ® m G) = (ft m _i F) ®| (ft m _j G) 

Occurrences under l m can only be recursive types, never parameters, so we always inject 
them on the right for lj. 

We can also show the equivalence between the interpretations. We first need a way 
to lift the indexing function from multirec to indexed. We do this by applying the 
same function on the right — on the left we have zero parameters, so we match on the 
absurd pattern: 

t_ : {I : Set} -» Indexed™ I -> Indexed (1 W I) 

(T 0 (InJi ()) 

(T r) (inj 2 I) = r i 

The equivalence of the interpretations (without considering fixed points) follows: 



Tm- 


i : {I 


Set} {r : lndexed m 1} {i : 1} 






(C 


: Code m 1) -» [Cl m ri = [ tm- 




Tm- 


i u m 


= refl 




Tm- 


i (lm I) 


= refl 




Tm- 


i (lm I) 


= refl 




Tm- 


i (F© m 


G) = cong 2 _W_ (t m _n F) (Tm^ 


|G) 


Tm- 


i (F® m 


G) = cong 2 _x_(t m ^i F) (T m _ 


|G) 



We omit the proof with fixed points, since it relies on the same techniques used in the 
following section, where we show the proofs in more detail. 



8.2.5 Indexed to InstantGenerics 

We now show how to convert from a fixed-point view to the coinductive representation 
of instant-generics (subscript ig). Since all fixed-point views embed into indexed, 
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we need to define only the embedding of indexed into instant-generics. Since the 
two universes are less similar, the code transformation reguires more care: 



fTi^g : {10 : Set} 

flUig U| r 0 = Uig 

flUig (li i) ro = r i 

Ib-iB (III) ro = K ig 

fTi^g (F ©i G) r 0 = (fTi^g 

ft-ig (F®,G) ro = (t 
fTi^ig (F@i G) ro 

(Fix, F) ro = R ig (ft fTi-ig I 

tf3 ig : {10 : Set} -> Code: I O 
ItPiigCro = tTi-ig C (K ig o r) o 



Codei 10 — >- (I -> Code ig ) -> (O ->■ Code ig ) 



0 = 1) 

Fro) 
ig F r o) 



® ig (fTi-ig G r o) 
®ig (1h- 



G ro) 



Rig (tt fri-»ig F (fr^ig G r) o) 



>, fTi-ig (Fix, F) r]o) 
(I ->• Set) ->• (O ->• Code iE 



We use two code conversion functions, ftfjg is the user-visible function, since it takes 
a parameter r that converts input indices into Sets. However, during the conversion we 
need to know how input indices map into Codei g s; for this we use the ti^ig function. 

Unit, sum, and product exist in both universes, so their conversion is trivial. Recursive 
invocations with lj are handled by r, which tells how to map indices to Codei g s. We 
lose the ability to abstract over recursive positions, which is in line with the behavior 
of instant-generics. Tagging is converted to a constant, trivially inhabited if we are 
in the expected output index o, and empty otherwise. Note that since an indexed code 
can define multiple types, but an instant-generics code can only represent one type, 
■ffj-.ig effectively produces multiple instant-generics codes, one for each output index 
of the original indexed family. 

A composition F ®j G is encoded through recursion; the resulting code is the conver- 
sion of F, whose parameters are indexed G functors. A fixed-point Fix, F is naturally 
encoded through recursion, in a similar way to composition. The recursive positions of 
the fixed point are either: parameters on the left, converted with r as before; or recursive 
occurrences on the right, handled by recursively converting the codes with "fti—ig and 
interpreting. 

Having the code conversion in place, we can proceed to convert values: 



from 


->ig 


: {I 0 : Set} {r : 






(C : Codei I 0) ( 


from 


->ig 


Ui o tt 


from 


— ig 


(li i) ox 


from 


->ig 


(!,i) ox 


from 


^ig 


(F©| G) o (inji x) = 


from 


^ig 


(F ©i G) o (inj 2 x) = 


from 


— ig 


(F ®i G) o (x , y) = 


from 


^ig 


(F®iG)ox 


from 


->ig 


(Fix: F) o ( x }| = 



-y Code ig } 
O) -> [C] 



o r o 



ffWig C ro] 



r\,g A 

inji ig 



(frorrij. 



+ ig 
Fox 



Fox) 
G o x) 
) ,i g (frami- 



ng 1 " A ; .ig v lull| i^ig G 0 y) 
reCjg (from^ig F o (mapj F (from^ G) o x)) 
rec ig (framing F o (mapi F [ id^i , fron-i^g (Fix, F) ] o x)) 
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fromfjfig : {I O : Set} (C : Code; I O) (r : I Set) (o : O) 

[Cl.ro->! trfiigCroJig 
from^ig C r o = fromj_>j g {r = K ig o r} C o o map. C (A _ — > k ig ) o 

Again we use two functions to account for the fact that user datatypes know only how 
to map input indices into Sets, and not into Codei g s. Function from^g invokes fromj_,j g 
after performing the necessary wrapping of indices with kj g . The cases for composition 
and fixed point are the most interesting because we have to map the conversion function 
inside the argument positions; we do this using the map, function. As usual, the inverse 
function tOi_> ig is entirely symmetrical, so we omit it. 

Note that in the type of fromj_>jg we instantiate the function which interprets in- 
dices (the r argument) with an instant-generics interpretation. However, r has type 
I — » Set, whereas J_]j g has return type Set-|. If we were to raise indexed to Seti, the 
interpretation function would then have type I — » Seti, but then we could no longer 
use it in the I, case. For now we rely on the Agda flag — type-in-type, and leave a 
formally correct solution for future work (see |Section 8.4} . 

It remains to show that the conversion functions form an isomorphism. We start with 
the two most interesting cases: composition and fixed points. Following jSection 6.2 
we use lifted composition, eguality, and identity to natural transformations in indexed 
(respectively _o- Ji _ J _= =?i _, and id^i). We use eguational reasoning for the proofs: 

isoi^ig : {10 : Set} (C : Codei I O) 

(r : I -> Code ig ) -> (to^g {r = r}Co^ from,^ C) id^; 

isoi^ig (F©i G) r o x = 

begin 

map, F (tOj_Hg G) o (tOj_>j g F o (frc-m^g F o (map, F (frc-m^g G) o x))) 
=( cong (map; F (to^g G) o) (ison^ig F _ o _) } 

map, F (to^ig G) o (map, F (frc-m^g G) o x) 
=( sym (map° F (tOi_>i g G) (framing G) o x) } 

map, F (tO|_»ig G frc-m^g G) o x 
=( map? F (ison-jig G r) o x } 

map, F id^i o x 
=( map| d Fox) 

xl 

The proof for composition is relatively simple, relying on applying the proof recursively, 
fusing the two maps, and reasoning by recursion on the resulting map, which results in 
an identity map. The proof for fixed points is slightly more involved: 

isoi^g (Fix, F) r o (x), = cong (_), $ 
begin 

map, F [ id^i , tOi^ ig (Fixj F) ] o 
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(tOi_> ig F o (framing F o (map, F [ icU^ , from,^ (Fix, F) ] o x))) 
=( cong (map, F [ id^i , tOi^ ig (Fix, F) ] o) (isoi j_>j g F _ o _) ) 

map, F [ id^i , toi^ ig (Fix, F) ] o (map, F [ id^i , from,^ (Fixj F) ] o x) 
=( sym (map° F [ id =ti , tOi_ ig (Fixj F) ] [ id^ ; , frani^g (Fix, F) ] o x) } 



fid- 



mapi F ([ id^i , to^g (Fixj F) ] o 
e( sym (map- F [.Jo, o x) } 
map, F [id^i o-^. id^i , tOi^ ig (Fix, F) o =f| from 



from^ig (Fix, F) ]) o x 



(Fixj F) ] o x 
refl) (ison-.ig (FiXi F) r)) o x ) 



=( map- F ([.Jcong, (A 

map, F [ id^i , id^i ] o x 
=( ma$ F [,]idi o x } 

map, F id^i o x 
=( mapj d Fox) 

xl 

We start in the same way as with composition, but once we have fused the maps we have 
to deal with the fact that we are mapping distinct functions to the left (arguments) and 
right (recursive positions). We proceed with a lemma on disjunctive maps that states 
that a composition of disjunctions is the disjunction of the compositions ([,]°j). Then we 
are left with a composition of identities on the left, which we solve with reflexivity, and 
a composition of to^ig and from^ig on the right, which we solve by induction. Finally, 
we show that a disjunction of identities is the identity (with the [,]idi lemma), and that 



the identity map is the identity. Note that we use the functor laws from Section 6.2 
whose types are repeated here for convenience: 



map" 



map" 



map. 



{10: Set} {r s : Indexedi 1} {f g : i 
f ==$; g -> mapi C f mapi C g 
{10: Set} {r s t : Indexed; 1} (C : Code, I O) (f 
map, C (f o-j, g) = ->. (map, C f o-j, map, C g) 

{10: Set} {r : Indexedi 1} (C : Code; I O) ->• 
map, {r = r} C id^; id^; 



s} (C : Codei I O) 
t) (9 



Note also that [,]o,, [,]idj, and [,]cong; are similar to ||o 1( ||idi, and Hcong, from Section 6.2 

respectively, but stated in terms of [_,_] instead of f | j . 

The proof for the remaining codes is unsurprising: 



isoii- 


.ig U, 




r o 




= refl 


isoi 


-g (li 0 




r o 




= refl 


isoii- 


^g Oi i) 




r o 




= refl 


isoi i_ 


.ig (F©i 


G) 


r o 


(inj-i x) 


= cong inji (ison^ig F r o x) 


isoi i_ 


.ig (F©i 


G) 


r o 


(inj 2 x) 


= cong inj 2 (isoij_>i g G r o x) 


iSO! 


.ig (F®, 


G) 


r o 


(x.y) 


= cong 2 _,_ (iso-i i^ ig F r o x) (iso-i i^ ig 



G r o y) 
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8.3 Applying generic functions across approaches 

The main operational advantage of our comparison is that it allows us to apply generic 
functions from approach B to representations of approach A, as long as A embeds into B. 
In this section we show some examples of that functionality. 

InlSection 6.1 .3|we introduced an indexed code 'ListF' for lists (which we name 'ListF', 



in this section). Using that code we can define an example list in indexed: 
aListi : ['ListF, J, (const T) tt 

aListi = ( inj 2 (tt , ( inj 2 (tt , ( inj 2 (tt , ( lnj 1 tt ),) ),) ),) ), 
This encodes the exact same list as aListjg from Section 7.1 We can then apply the 



sizejg function (defined in the same section) to aListi, after converting using the functions 
shown in ISection 8.2.51 

testing : size ig (flf^'g 'ListF, _ _) (fromf^g 'ListF', _ _ aListi) = 4 
testing = refl 

Recall that sizejg takes two arguments: first the code, which is the result of converting 
the list code 'ListF', to instant-generics with jlSV and then the value, which we 
convert with framing. The parameters regarding indexed indices can be figured out by 
Agda, so we replace them by an underscore. 



Recall now the Zig/Zag family of mutually recursive datatypes introduced in Sec- 



tion 5.1 We have defined many generic functions in the indexed universe, and argued 



Chapter 6 that these could be applied to the other approaches shown so far. For 



instance, we can apply decidable eguality to the zigZagEnd value: 
tesC^i : Maybe („ = _) 

tesCii = deqt; (Fbq (tf m _n ZigZagCJ) (A _ _ -> A ()) _ x x 
where x = from^ m ^j zigZagEnd m 

testae : test^ = just refl 
test m _,i = refl 

Note also that the embedding relation is transitive. The following example brings 
the encoding of a natural number aNat from regular {Section 3.1 1 into indexed, and 
traverses it with a catamorphism, effectively doubling the number: 

test^ : N 

testS = cata i - f - (from/7p^i _ (from^ r ^ p _ aNat r )) 
where f : T — > T W N — > N 
f _ (inj! _) = 0 
f _ (inj? n) = 2 + n 

test r ^i : testS = 4 
tesWi = refl 

The ability to apply generic functions across approaches means that the functions only 
have to be defined in one universe, and can be applied to data from any embeddable 



92 



8.4 Discussion 



universe. This is particularly relevant for advanced generic behaviour, such as the 
zipper, which can be complicated to define in each universe. In effect, by defining an 
indexed zipper in Section 6~4| we have also defined a zipper for the regular, polyp, 
and multirec approaches, using the code conversions defined in this chapter. 



8.4 Discussion 

In this chapter we have compared different generic programming universes by showing 
the inclusion relation between them. This is useful to determine that one approach 
can encode at least as many datatypes as another approach, and also allows for lifting 
representations between compatible approaches. This also means that generic functions 
from approach B are all applicable in approach A, if A embeds into B, because we can 
bring generic values from approach A into B and apply the function there. Efficiency 
concerns remain to be investigated and addressed. 

Our comparison does not allow us to make statements about the variety of generic 
functions that can be encoded in each approach. The generic map function, for instance, 
cannot be defined in instant-generics, while it is standard in indexed. One possible 
direction for future research is to devise a formal framework for evaluating what generic 
functions are possible in each universe, adding another dimension to our comparison of 
approaches. 

Notably absent from our comparison are libraries with a generic view not based on a 



sum of products. In particular, the spine view Hinze et a I. 2006 is radically different 
from the approaches we model; yet, it is the basis for a number of popular generic 
programming libraries. It would be interesting to see how these approaches relate to 
those we have seen, but, at the moment, converting between entirely different universes 
remains a challenge. 

An issue that remains with our modelling is to properly address termination. While 
our conversion functions can be used operationally to enable portability across different 
approaches, to serve as a formal proof they have to be terminating. Since Agda's al- 
gorithm for checking termination is highly syntax-driven, attempts to convince Agda of 
termination are likely to clutter the model, making it less easy to understand. We have 
thus decided to postpone such efforts for future work, perhaps relying on sized types for 



guaranteeing termination of our proofs Abel 2010 



>ec- 



A related issue that remains to be addressed is our use of — type-in-type in 
Ition 8.2.51 for the code conversion function. It is not clear how to solve this issue even 
with the recently added support for universe polymorphism in Agda. 

Nonetheless, we believe that this work is an important first step towards a formal 
categorisation of generic programming libraries. Future approaches can rely on our 
formalisation to describe precisely the new aspects they introduce, and how the new 
approach relates to existing ones. In this way we can hope for a future of modular 
generic programming, where a specific library might be constructed using components 
from different approaches, tailored to a particular need while still reusing code from 
existing libraries. 
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Practical aspects of generic 
programming 
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CHAPTER 9 



Optimisation of generic programs 



In the first part of this thesis we have introduced multiple libraries for generic program- 
ming in Haskell. In Chapter 8 we have compared them in terms of expressiveness, but we 
have not looked at how fast they perform. It is widely believed that generic programs run 
slower than type-specific handwritten code, given the conversions to and from represen- 
tation types, and this factor often prevents adoption of generic programming altogether. 
On the other hand, generic functions can be specialised to particular datatypes, remov- 
ing any overhead from the use of generic programming. In this chapter we analyse how 
to improve the performance of generic programs in GHC. We choose a representative 
library and look at the generated core code for a number of example generic functions. 
After understanding the necessary compiler optimisations for producing efficient generic 
code, we benchmark the runtime of our generic functions against handwritten variants, 
and conclude that all the overhead can indeed be removed. 



9.1 Introduction 



The performance of generic programs has been analysed before. |Rodriguez Yakushev 
et a I. |2008 present a detailed comparison of nine libraries for generic programming, 
with a brief performance analysis. This analysis indicates that the use of a generic 
approach could result in an increase of the running time by a factor of as much as 80. 



Van Noort et a I. 2010 also report severe performance degradation when comparing a 



generic approach to a similar but type-specific variant. While this is typically not a 
problem for smaller examples, it can severely impair adoption of generic programming 
in larger contexts. This problem is particularly relevant because generic programming 
technigues are especially applicable to large applications where performance is crucial, 
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such as structure editors or compilers. 

To understand the source of performance degradation when using a generic function 
from a particular generic programming library, we have to analyse the implementation 
of the library. The fundamental idea behind generic programming is to represent all 
datatypes by a small set of representation types. Eguipped with conversion functions 
between user datatypes and their representation, we can define functions on the repre- 
sentation types, which are then applicable to all user types via the conversion functions. 
While these conversion functions are typically trivial and can be automatically generated, 
the overhead they impose is not automatically removed. In general, conversions to and 
from the generic representations are not eliminated by compilation, and are performed 
at run-time. These conversions are the source of inefficiency for generic programming 
libraries. In the earlier implementations of generic programming as code generators or 



preprocessors Hinze et a I. 2007, optimisations (such as automatic generation of type- 
specialised variants of generic functions) could be implemented externally. With the 
switch to library approaches, all optimisations have to be performed by the compiler, as 
the library approach no longer generates code itself. 

GHC compiles a program by first converting the input into a core language and then 
transforming the core code into more optimised versions, in a series of seguential passes. 
While it performs a wide range of optimisations, with the default settings it seems to 
be unable to remove the overhead incurred by using generic representations. Therefore 



generic libraries perform slower than handwritten type-specific counterparts. Alimarine 



and Smetsers 2004 2005 show that in many cases it is possible to remove all overhead 
by performing a specific form of symbolic evaluation in the Clean language. In fact, 
their approach is not restricted to optimising generics, and GHC performs symbolic 
evaluation as part of its optimisations. Our goal is to convince GHC to optimise generic 
functions so as to achieve the same performance as handwritten code, without reguiring 
any additional manipulation of the compiler internals. 

We have investigated this problem before |Magalhaes et al" 2010b, and concluded 



that tweaking GHC optimisation flags can achieve significant speedups. The problem 
with using compiler flags is that these apply to the entire program being compiled, and 
while certain flags might have a good effect on generic functions, they might adversely 
affect performance (or code size) of other parts of the program. In this chapter we take a 
more fine-grained approach to the problem, looking at how to localise our performance 
annotations to the generic code only, by means of rewrite rules and function pragmas]^] 
In this way we can improve the performance of generic functions with minimal impact on 
the rest of the program. 

We continue this chapter by defining two representative generic functions which we 
focus our optimisation efforts on {Section 9.2) . We then see how these functions can 
be optimised ma nually {Section 9.3) , and transfer the necessary optimisation technigues 
to the compiler {Section 9.4) . We confirm that our optimisations result in better run- 



time performance of generic programs in a benchmark in Section 9.5 and conclude in 
ISection 9.6I 



1 http: //www.haskell . org/ghc/docs/latest/html/users_guide/pragmas .html 
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9.2 Example generic functions 



For analysing the performance of generic programs we choose a simple but representative 
generic programming library from those presented in Part l| na mely instant-generics 
jChapterT . We reuse the Haskell encoding of Section 7.2 and present two generic 
functions that will be the focus of our attention: eguality and enumeration. 



9.2.1 Generic equality 

A notion of structural eguality can easily be defined as a generic function. We start by 
giving the user-facing class and eguality method: 

class GEq a where 

geq :: a — > a -* Bool 

This class is similar to the Prelude Eq class, but we have left out ineguality for simplicity. 
Adhoc instances for base types can reuse the Prelude implementation: 

instance GEq Int where 

geq = (=) 

To be able to give generic instances for types such as lists, we first define a class for 
equality on the representation types: 

class GEq Rep a where 

geqRep :: a — > a -> Bool 

We can now give instances for each of the representation types: 

instance GEqRep U where 
geqRep = True 

instance (GEqRep a, GEqRep /?) => GEqRep (a :+: /?) where 
geqRep (L x) (L y) = geqRep x y 
geqRep (R x) (R y) = geqRep x y 
geqRep = False 

instance (GEqRep a, GEqRep (3) => GEqRep (a :x: /?) where 
geqRep fa :x: (x 2 :x: y 2 ) = geqRep xi x 2 A geqRep y, y 2 

Units are trivially equal. For sums we continue the comparison recursively if both values 
are either on the left or on the right, and return False otherwise. Products are equal if 
both components are equal. 

For variables and recursion we use the GEq class we have defined previously: 

instance (GEq a) => GEqRep (Rec a) where 
geqRep (Rec x) (Rec y) = geq x y 
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instance (GEq a) => GEqRep (Var a) where 
geqRep (Var x) (Var y) = geq x y 

The generic eguality function, to be used when defining the geq method for generic 
instances, simply applies geqRep after converting the values to their generic represen- 
tation: 

geqDefault :: (Representable a, GEqRep (Rep a)) => a -* a — > Bool 
geqDefault x y = geqRep (from x) (from y) 

Now we can give a generic instance for lists: 

instance (GEq a) => GEq [ a ] where 
geq = geqDefault 

9.2.2 Generic enumeration 

We now define a function that enumerates all possible values of a datatype. For infinite 
datatypes we have to make sure that every possible value will eventually be produced. 
For instance, if we are enumerating integers, we should not first enumerate all positive 
numbers, and then the negatives. Instead, we should interleave positive and negative 
numbers. 

While eguality is a generic consumer, taking generic values as input, enumeration is 
a generic producer, since it generates generic values. We enumerate values by listing 
them with the standard list type. There is only one unit to enumerate, and for variables 
and recursion we refer to the user-facing GEnum class as usual: 

class GEnumRep a where 



instance GEnumRep U where 
genumRep = [U] 

instance (GEnum a) => GEnumRep (Rec a) where 
genumRep = map Rec genum 

instance (GEnum a) => GEnumRep (Var a) where 
genumRep = map Var genum 

The more interesting cases are those for sums and products. For sums we enumerate 
both alternatives, but interleave them with a (|||) operator: 

instance (GEnumRep a, GEnumRep f3) => GEnumRep (a :+: /?) where 
genumRep = map L genumRep ||| map R genumRep 



genumRep :: [ a 
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For products we generate all possible combinations of the two arguments, and diago- 
nalise the result matrix, ensuring that all elements from each sublist will eventually be 
included, even if the lists are infinite: 

instance (GEnumRep a, GEnumRep f3) => GEnumRep (a :x: f3) where 
genumRep = diag (map (\x — > map (\y — > x :x: y) genumRep) genumRep) 

diag ::[[»]] — > [a] 

We omit the implementation details of (|||) and diag as they are not important; it only 
matters that we have some form of fair interleaving and diagonalisation operations. The 
presence of (|||) and diag throughout the generic function definition makes enumeration 
more complicated than eguality, since eguality does not make use of any auxiliary 
functions. We will see in Section 9.4.3 how this complicates the specialisation process. 



Note also that we do not use the more natural list comprehension syntax for defining 
the product instance, again to simplify the analysis of the optimisation process. 
Finally we define the user-facing class, and a default implementation: 

class GEnum a where 
genum ::[a] 

genumDefault :: (Representable a, GEnumRep (Rep a)) => [a] 
genumDefault = map to genumRep 



9.3 Specialisation, by hand 

We now focus on the problem of specialisation of generic functions. By specialisation 
we mean removing the use of generic conversion functions and representation types, 
replacing them by constructors of the original datatype. To convince ourselves that this 
task is possible we first develop a hand-written derivation of specialisation by eguational 
reasoning. For simplicity we ignore implementation mechanisms such as the use of type 
classes and type families, and focus first on a very simple datatype encoding natural 
numbers: 

data Nat = Ze | Su Nat 

We give the representation of naturals using a type synonym: 

type RepNat = U :+: Nat 

We use a shallow representation (with Nat at the leaves, and not RepNat), remaining 
faithful with instant-generics. We also need a way to convert between RepNat and 
Nat: 
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toNat :: RepNat ->■ Nat 
toNat n = case n of 

(L U) ->■ Ze 

(R n) -> Su n 

fromNat :: Nat -» RepNat 
from Nat n = case n of 

Ze LU 

(Su n) ^ Rn 

We now analyse the specialisation of generic eguality and enumeration on this datatype. 
9.3.1 Generic equality 

We start with a handwritten, type-specific definition of eguality for Nat: 

eqNat :: Nat -> Nat -» Bool 
eqNat m n = case (m , n) of 

(Ze , Ze) — ► True 

(Su m , Su n) —> eqNat m n 

(„ , _) False 

For equality on RepNat, we need eguality on units and sums: 

eqU :: U -»• U -* Bool 

eqLI x y = case (x , y) of 

(U , U) -> True 

eqPlus :: (a -> a -> Bool) -> (j8 -» jS -» Bool) -> 

a :+: |8 -> a :+: j8 -> Bool 

eqPlus ea eb a b = case (a , b) of 

(L x , L y) — > ea x y 
(R x , R y) — > eb x y 
(_ , _) False 

Now we can define equality for RepNat, and generic eguality for Nat through conversion 
to RepNat: 

eqRepNat :: RepNat -> RepNat -> Bool 
eqRepNat = eqPlus eqU eqNatFromRep 

eqNatFromRep :: Nat -> Nat -* Bool 

eqNatFromRep m n = eqRepNat (fromNat m) (fromNat n) 

Our goal now is to show that eqNatFromRep is equivalent to eqNat. In the following 
derivation, we start with the definition of eqNatFromRep, and end with the definition of 
eqNat: 
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eqRepNat (fromNat m) (fromNat n) 
=( inline eqRepNat } 

eqPlus eqU eqNatFromRep (fromNat m) (fromNat n) 

=( inline eqPlus } 

case (fromNat m , fromNat n) of 
(L x , L y) — > eqU x y 
(R x , R y) — > eqNatFromRep x y 
-» False 

=( inline fromNat } 

case ( case m of {Ze — > L U; Su x-t — > R x-^ } 
, case n of {Ze -» L U; Su x 2 -> R x 2 }) of 
(L x , L y) — > eqU x y 
(R x , R y) — > eqNatFromRep x y 
—> False 

=( case-of-case transform } 

case (m , n) of 

(Ze , Ze) -» eqU U U 
(Su X! , Su x 2 ) -* eqNatFromRep x-\ x 2 
— > False 

=( inline eqU and case-of-constant } 

case (m , n) of 

(Ze , Ze) -» True 
(Su X! , Su x 2 ) -* eqNatFromRep Xi x 2 
-» False 

=( inline eqNatFromRep , induction ) 

case (m , n) of 

(Ze , Ze) -* True 
(Su X! , Su x 2 ) -* eqNat X\ x 2 
-» False 



This shows that the generic implementation is equivalent to the type-specific variant, 
and that it can be optimised to remove all conversions. We discuss the techniques 
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used In this derivation In more detail In Section 9.4.1 after showing the optimisation of 
generic enumeration. 

9.3.2 Generic enumeration 

A type-specific enumeration function for Nat follows: 

enumNat :: [ Nat] 

enumNat = [Ze] ||| map Su enumNat 

To get an enumeration for RepNat we first need to know how to enumerate units and 
sums: 

enumLI :: [ U] 
enumLI = [U] 

enumPlus :: [ a } -> [ £ ] -» [ a :+: } 
enumPlus ea eb = map L ea ||| map R eb 

Now we can define an enumeration for RepNat: 

enumRepNat :: [ RepNat] 

enumRepNat = enumPlus enumLI enumNatFromRep 

With the conversion function toNat, we can use enumRepNat to get a generic enumeration 
function for Nat: 

enumNatFromRep :: [ Nat ] 

enumNatFromRep = map toNat enumRepNat 

We now show that enumNatFromRep and enumNat are egulvalent: 

map toNat enumRepNat 
= ( Inline enumRepNat } 

map toNat (enumPlus enumU enumNatFromRep) 
= ( Inline enumPlus } 

map toNat (map L enumU ||| map R enumNatFromRep) 
= ( Inline enumU } 

map toNat (map L [U] ||| map R enumNatFromRep) 
= ( Inline map } 
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map toNat ([L U] ||| map R enumNatFromRep) 
=( free theorem (|||) : V f a b. map f (a ||| b) = map f a ||| map f b ) 

map toNat [L U] ||| map toNat (map R enumNatFromRep) 
=( inline map } 

[toNat (L U)] HI map toNat (map R enumNatFromRep) 
=( inline toNat and case-of-constant ) 

[Ze] HI map toNat (map R enumNatFromRep) 
=( functor composition law: V f g I. map f (map g I) = map (f o g) I } 

[Ze] HI map (toNat o R) enumNatFromRep 
=( inline toNat and case-of-constant } 

[Ze] HI map Su enumNatFromRep 

Like equality, generic enumeration can also be specialised to a type-specific variant 
without any overhead. 



9.4 Specialisation, by the compiler 

After the manual specialisation of generic functions, let us now analyse how to convince 
the compiler to automatically perform the specialisation. 



9.4.1 Optimisation techniques 

Our calculations in |Section 9.3| rely on a number of lemmas and techniques that the 
compiler will have to use. We review them here: 



Inlining Inlining replaces a function call with its definition. It is a crucial optimisation 
technique because it can expose other optimisations. However, inlining causes code 
duplication, and care has to be taken to avoid non-termination through infinite inlining. 

GHC uses a number of heuristics to decide when to inline a function or not, and 
loop breakers for preventing infinite inlining Peyton Jones and Marlow 2002. The 



programmer can provide explicit inlining annotations with the INLINE and NOINLINE 
pragmas, of the form: 
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{-# INLINE [n] f #-} 

In this pragma, f is the function to be inlined, and n is a phase number. GHC performs 
a number of optimisation phases through a program, numbered in decreasing order until 
zero. Setting n to 1, for instance, means "be keen to inline f in phase 1 and after". For 
a NOINLINE pragma, this means "do not inline f in phase 1 or after". The phase can be 
left out, in which case the pragma applies to all phasesF] 



Application of free theorems and functor laws Free theorems Wadler 1989 are the 



orems that arise from the type of a polymorphic function, regardless of the function's 
definition. Each polymorphic function is associated with a free theorem, and functions 



with the same type share the same theorem. The functor laws (mentioned in Section 6.2 
for the indexed universe) arise from the categorical nature of functors. Every Functor 
instance in Haskell should obey the functor laws. 

GHC does not compute and use the free theorem of each polymorphic function, also 
because it may not be clear which direction of the theorem is useful for optimisation 
purposes. However, we can add special optimisation rules to GHC via a RULES pragma 
|Peyton Jones et"aL 2001 . For instance, the rewrite rule corresponding to the free 



theorem of (III) follows: 



{-# RULES "ft Ml" forall f a b. map f (a ||| b) = map f a ||| map f b #-} 

This pragma introduces a rule named "ft |||" telling GHC to replace appearances of the 
application map f (a ||| b) with map f a ||| map f b. GHC does not perform any confluence 
checking on rewrite rules, so the programmer should ensure confluence or GHC might 
loop during compilation. 



Optimisation of case statements Case statements drive evaluation in GHC's core lan- 



guage, and give rise to many possible optimisations. Peyton Jones and Santos 1998 



provide a detailed account of these; in our derivation in Section 9.3.2 we used a "case 
of constant" rule to optimise a statement of the form: 



case (L ()) of 

L 0 -» Ze 
R n -> Su n 



Since we know what we are case-analysing, we can replace this case statement by the 
much simpler expression Ze. Similarly, in |Section 9.3.1 | we used a case-of-case transform 
to eliminate an inner case statement. Consider an expression of the form: 

case (case x of {e-i -* e 2 }) of 
e 2 -> e 3 

2 See the GHC User's Guide for more details: |http : //www . haskell . org/ ghc/ docs/latest /html/users_ 
e/pr agmas . html. 
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Here, e-i, e2, and e3 are expressions starting with a constructor. We can simplify this to: 



case x of 

ei ->■ e 3 



This rule naturally generalises to case statements with multiple branches. 



9.4.2 Generic equality 

We have seen that we have a good number of tools at our disposal for directing the 
optimisation process in GHC: inline pragmas, rewrite rules, phase distinction, and all 
the standard optimisations for the functional core language. We will now annotate our 
generic functions and evaluate the guality of the core code generated by GHC. 
We start by defining a Representable instance for the Nat type: 

instance Representable Nat where 
type Rep Nat = U :+: Rec Nat 

to (L U) = Ze 

to (R (Rec n)) = Su n 

from Ze = LU 
from (Su n) = R (Rec n) 

We can now provide a generic definition of eguality for Nat: 



instance GEq Nat where 

geq = geqDefault 

Compiling this code with the standard optimisation flag -0, and using -ddump-simpl 
to output the generated core code after all the simplification passes, we get the following 
core code: 



$GEqNat geq :: Nat -> Nat ->• Bool 

$GEqNat geq = A (x :: Nat) (y :: Nat) -> case x of 



Ze — > case y of 

Ze -* True 

Sum False 
Su m -* case y of 

Ze -* False 

Su n -» $GEqNat gep m n 



The core language is a small, explicitly typed language in the style of System F Yorgey 
The function $GEqNat geq is prefixed with a $ because it was generated by 



et a I 



2012 



the compiler, representing the geq method of the GEq instance for Nat. We can see that 
the generic representation was completely removed, even without any INLINE pragmas. 
The same happens for lists, as evidenced by the generated core code: 
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$GEq[] geq :: forall a. GEq a => [ a] — > [ a] — > Bool 
$GEq[] geq = A a (eqA :: GEq a) (h :: [ a ]) (l 2 :: [ a ]) 
case h of 

-* case l 2 of 

— > True 
(h : t) ->• False 
(hi! : ti) -» case l 2 of 

[] -> False 
(h 2 : t 2 ) -> case eqA hi! h 2 of 
False -» False 

True -* $GEq[] geq a eqA ti t 2 

Note that type abstraction and application is explicit in core. There is syntax to distin- 
guish type and value application and abstraction from each other, but we suppress the 
distinction since it is clear from the colour. Note also that constraints (to the left of the 
=> arrow) become just ordinary parameters, so $GEq[] gep takes a function to compute 
equality on the list elements, eqA^j 



Perhaps surprisingly, GHC performs all the required steps of Section 9.3.1 without 
requiring any annotations. In general, however, we found that it is sensible to pro- 
vide INLINE pragmas for each instance of the representation datatypes when defining a 
generic function. In the case of geqRep, the methods are small, so GHC inlines them 
eagerly. For more complicated generic functions, the methods may become larger, and 
GHC will avoid inlining them. Supplying an INLINE pragma tells GHC to inline the 
methods anyway. The same applies to the from and to methods; for Nat and lists these 
methods are small, but in general they should be annotated with an INLINE pragma. 



9.4.3 Generic enumeration 

Generic consumers, such as equality, are, in our experience, more easily optimised by 
GHC. A generic producer such as enumeration, in particular, is challenging because it 
requires map fusion, and lifting auxiliary functions through maps using free theorems. As 
such, we encounter some difficulties while optimising enumeration. We start by looking 
at the natural numbers: 

instance GEnum Nat where 
genum = map to genumRep 

Note that instead of using genumDefault we directly inline its definition; this is to 
circumvent a limitation in the current implementation of defaults that prevents later 
rewrite rules from applying. GHC then generates the following code: 

The type of eqA is GEq a, but we use it as if it had type a —> a —> Bool. In the yenerated core there 
is also a coercion around the use of eqA to transform the class type into a function. This is the standard 
way class methods are desuyared into core; we elide these details as they are not relevant to optimisation 
itself. 
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$x 2 :: [U :+: Rec Nat] 

$x 2 = map $x 4 $GEnumNatgenum 

$X! :: [U :+: Rec Nat] 

$X-| = $X3 HI $X 2 

$GEnumNatg enum ::[Nat] 
$GEnumNatgenum = map to 

We omit the definitions of $x 3 and $x 4 for brevity. To make progress we need to tell 
GHC to move the map to expression in $GEnumNat ge num through the (|||) operator. We 
use a rewrite rule for this: 

{-# RULES "ft Ml" forall f a b. map f (a ||| b) = map f a ||| map f b #-} 

With this rule in place, GHC generates the following code: 

$x 2 :: [U :+: Rec Nat] 

$x 2 = map $x 4 $GEnumNatgenum 

$X! :: [Nat] 

$X! = map to $x 2 

$GEnumNatgenum ::[Nat] 
$GEnumNatgenum = $* 3 III $ x i 

We now see that the $x-| term is map applied to the result of a map. The way map is 
optimised in GHC (by conversion to build/foldr form) interferes with our "ft IN" rewrite 
rule, and map fusion is not happening. We can remedy this with an explicit map fusion 
rewrite rule: 

{-# RULES " map/map 1" forall f g I. map f (map g I) = map (f o g) I #-} 

This rule results in much improved generated code: 

$x 3 :: [U :+: Rec Nat] 
$x 3 = $x 4 : [] 

$x 2 :: [Nat] 

$x 2 = map to $x 3 

$Xi :: [Nat] 

$X! = map Su $GEnumNatgenum 
$GEnumNatgenum ::[Nat] 
$GEnumNatgenum = $x 2 ||| 

The only thing we are missing now is to optimise $x 3 ; note that its type is [ U :+: Rec Nat], 
and not [Nat]. For this we simply need to tell GHC to eagerly map a function over a 
list with a single element: 

{-# RULES "map/map2" forall f x. map f (x : []) = (fx): [] #-} 
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With this, GHC can finally generate the fully specialised enumeration function on Nat: 

$x 2 :: [Nat] 
$x 2 = Ze:[] 
$Xi :: [Nat] 

$Xi = map Su $GEnumNatgenum 
SGEnumNatgenum " [Nat] 
SGEnumNatgenum = $x 2 ||| $ X1 

Optimisation for lists proves to be more difficult. Since lists use products, we need to 
introduce a rewrite rule for the free theorem of diag, allowing map to be pushed inside 
diag: 

{-# RULES "f t/diag" forall f I. map f (diag I) = diag (map (map f) I) #-} 

With this rule, and the extra optimisation flag -f no-full-laziness to maximise the 
chances for rewrite rules to apply, we get the following code: 

$GEnum[] genum :: forall a. GEnum a => [[a]] 
$GEnum[] genum = A (gEnumA :: GEnum a) —> 
([]:[]) HI let $ X1 ::[Rec[a]] 

$Xi = map Rec ($GEnum[] genum gEnumA) 
in diag (map (A ($x 3 :: a] -* 

map (A ($x 2 :: Rec [«])-> case $x 2 of 

Rec $x 4 -» $x 3 : $x 4 ) $ X1 ) 

gEnumA) 

Most of the generic overhead is optimised away, but one problem remains: $Xi maps 
Rec over the recursive enumeration elements, but this Rec is immediately eliminated 
by a case statement. If $Xi was inlined, GHC could perform a map fusion, and then 
eliminate the use of Rec altogether. However, we have no way to specify that $Xi should 
be inlined; the compiler generated it, so only the compiler can decide when to inline it. 
Also, we had to use the compiler flag -f no-full-laziness to prevent some let-floating, 
but the flag applies to the entire program and might have unintended side-effects. 
Reflecting on our developments in this section, we have seen that: 

• Convincing GHC to optimise genum for a simple datatype such as Nat reguires 
the expected free theorem of (|||). However, due to interaction between phases of 
application of rewrite rules, we are forced to introduce new rules for optimisation 
of map. 

• Optimising genum for a more complicated datatype like lists reguires the expected 
free theorem of diag. However, even after further tweaking of optimisation flags, 
we are unable to derive a fully optimised implementation. In any case, the partial 
optimisation achieved is certainly beneficial. 
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• More generally, we see that practical optimisation of generic functions Is hard 
because of subtle Interactions between the different optimisation mechanisms In- 
volved, such as Inllnlng, rewrite rule application, let floating, case optimisation, 
etc. 

These experiments have been performed with the latest GHC version available at the 
time of writing, 7.4.1. We have also observed that the behavior of the optlmlser changes 
between compiler versions. In particular, some technlgues which resulted In better code 
In some versions (e.g. the use of SPECIALISE pragmas) result In worse code In other 
versions. This makes It practically Impossible to devise a set of general guidelines for 
generic function optimisation; unfortunately, users regulrlng the best performance will 
have to Inspect the generated core code and experiment with the possible optimisations. 

9.5 Benchmarking 

Having seen the Inner workings of the GHC optlmlser, we now verify experimentally 
that our optimisation technlgues result In a runtime Improvement of code using generic 
functions. 



9.5.1 Benchmark suite design 

Benchmarking Is, In general, a complex task, and a lazy language Imposes even more 
challenges on the design of a benchmark. We designed a benchmark suite that en- 
sures easy repeatability of tests, calculating the average running time and the standard 
deviation for statistical analysis. It Is portable across different operating systems and 
can easily be run with different compiler versions. It supports passing additional flags 
to the compiler and reports the flags used In the final results. Each test Is compiled 
as an Independent program, which consists of getting the current CPU time, running 
the test, getting the updated CPU time and outputtlng the time difference. The Unix 
time command performs a similar task, but unfortunately no egulvalent Is Immediately 
available In Microsoft Windows. By Including the timing code with the test (using the 
portable function System. CPUTime.getCPUtime) we work around this problem. 

To ensure reliability of the benchmark we use profiling, which gives us Information 
about which computations last longer. For each of the tests, we ensure that at least 
50% of the time Is spent on the function we want to benchmark. Profiling also gives 
Information about the time It takes to run a program, but we do not use those figures 
since they are affected by the profiling overhead. An approach similar to the nofib 



benchmark suite |Partaln 1993 Is not well-suited to our case, as our main point Is 
not to compare an Implementation across different compilers, but Instead to compare 
different Implementations on a single compiler. 

A top-level Haskell script takes care of compiling all the tests with the same flags, 
Invoking them a given number of times, parsing and accumulating results as each test 
finishes, and calculating and displaying the average running time at the end, along with 
some system Information. 
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Figure 9.1: Running time comparison between handwritten, generic, and optimised 
generic code for three independent tests. 



We have a detailed benchmark suite over different datatypes, generic functions, and 
generic programming libraries]^] Here we show only a small subset of the results gath- 
ered from our benchmark, to confirm the insights gained from Section 9.4 



9.5.2 Results 

We have tested generic eguality and enumeration on a simple datatype of binary trees: 
data Tree a = Bin a (Tree a) (Tree a) | Leaf 



For enumeration we have also used the Nat datatype. The results can be seen in 
|Figure 9~T We compare the running time for each of three variants: handwritten, 
instant-generics without any optimisations, and instant-generics with the opti- 
misations described previously, all compiled with -01. For eguality, we can see that 
using our optimisations removes all the overhead from generic programming, as we 
expected. Enumeration on Nat is also optimised to perfection, but for more complex 



datatypes, such as Tree, we run into the problems observed in Section 9.4.3 while the 
optimisations are beneficial, they are not enough to bring the performance to the same 
level as handwritten code. 



^https : //subversion, cs .uu.nl/repos/staf f . jpm.public/benchmark/trunk/ 
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9.6 Conclusion 

In this chapter we have looked at the problem of optimising generic functions. With their 
representation types and associated conversions, generic programs tend to be slower 
than their type-specific handwritten counterparts, and this can limit adoption of generic 
programming In situations where performance Is Important. We have picked one specific 
library, instant-generics, that Is representative of the type of generic programming 
we explore In this thesis, and Investigated the code generation for generic programs, and 
the necessary optimisation technlgues to fully remove any overhead from the library. We 
concluded that the overhead can be fully removed most of the time, using only already 
available optimisations that apply to functional programs In general. However, due to 
the difficulty of managing the Interaction between several different optimisations, In some 
cases we are not able to fully remove the overhead. We are confident, however, that this 
Is only a matter of further tweaking of GHC's optimisation strategies. 

Some work remains to be done In terms of Improving the user experience. We have 
mentioned that the to and from functions should be Inllned; this should be automatically 
established by the mechanism for deriving Representable Instances. Additionally, Insert- 
ing INLINE pragmas for each case In the generic function Is a tedious process, which 
should also be automated. Finally, It would be Interesting to see If the definition of 
rewrite rules based on free theorems of auxiliary functions used could be automated; It 
Is easy to generate free theorems, but It Is not always clear how to use these theorems 
for optimisation purposes. 

While instant-generics Is a representative library for the approaches covered In 



this thesis, other approaches, such as Scrap Your Boilerplate [SYB, Lammel and Pey- 



ton Jones 2003 2004, use different Implementation mechanisms and certainly regulre 
different optimisation strategies. It remains to be seen how to optimise other approaches, 
and to establish general guidelines for optimisation of generic programs. 

In any case, It Is now clear that generic programs do not have to be slow, and their 
optimisation up to handwritten code performance Is not only possible but also achievable 
using only standard optimisation technlgues. This opens the door for a future where 
generic programs are not only general, elegant, and concise, but also as efficient as 
type-specific code. 
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Generic programming for indexed datatypes 



An indexed datatype is a type that uses a parameter as a type-level tag; a typical 
example is the type of vectors, which are indexed over a type-level natural number 
encoding their length. Since the introduction of CADTs, indexed datatypes have become 
commonplace in Haskell. Values of indexed datatypes are often more involved than 
values of plain datatypes, and programmers would benefit from having generic programs 
on indexed datatypes. However, no generic programming library in Haskell adeguately 
supports them, leaving programmers with the tedious task of writing repetitive code. 

In this chapter we show how to extend the instant-generics approach to deal 
with indexed datatypes. Our approach can also be used in similar libraries, and is 
fully backwards-compatible, meaning that existing code will continue working without 
modification. We show not only how to encode indexed datatypes generically, but also 
how to instantiate generic functions on indexed datatypes. Furthermore, all generic 
representations and instances are generated automatically, making life easier for users. 

10.1 Introduction 

Runtime errors are undesirable and annoying. Fortunately, the strong type system of 
Haskell eliminates many common programmer mistakes that lead to runtime errors, like 
unguarded casts. However, even in standard Haskell, runtime errors still occur often. A 
typical example is the error of calling head on an empty list. 

Indexed datatypes, popular since the introduction of GADTs, allow us to avoid calling 
head on an empty list and many other runtime errors by encoding further information 
at the type level. For instance, one can define a type of lists with a known length, and 
then define head in such a way that it only accepts lists of length greater than zero. 
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This prevents the usual mistake by guaranteeing statically that head Is never called on 
an empty list. 

Unfortunately, generic programming and Indexed datatypes do not mix well. The 
added Indices and their associated type-level computations needs to be encoded In a 
generic fashion, and while this Is standard In dependently-typed approaches to generic 
programming (for Instance indexed of Chapter 6) , we know of no generic programming 
approach In Haskell that deals with Indexed datatypes. In fact, even the standard 
deriving mechanism, which automatically generates Instances for certain type classes, 
falls to work for GADTs, In general. 

We argue that It Is time to allow these two concepts to mix. Driven by an application 



that makes heavy use of both generic programming and Indexed datatypes Magalhaes 



and De Haas 2011 , we have developed an extension to instant-generics to support 



Indexed datatypes. Our extension Is both conservative, as It preserves all the function- 
ality of instant-generics without regulrlng modifications of client code, and general, 
as It applies egually well to other generic programming libraries. Furthermore, we show 
that Instantiating functions to Indexed datatypes Is not trivial, even In the non-generic 
case. In the context of datatype-generic programming, however, It Is essential to be able 
to easily Instantiate functions; otherwise, we lose the simplicity and reduced code du- 
plication we seek. Therefore we show how to automatically Instantiate generic functions 
to Indexed datatypes, In a way that works for most types of generic functions. 

Sec- 



We continue this chapter by first describing Indexed datatypes In detail 



tlon 10.2 Section 10.3 describes how Indexed datatypes can be represented generlcally, 



and 



Section 10.4 deals with Instantiating generic functions to these datatypes. In |Sec-| 



tlon 10.5 we sketch the algorithm for automatic representation of Indexed datatypes and 



Instantiation of generic functions. We then discuss how to deal with datatypes Indexed 
over custom kinds In Section 10.6 review related work In Section 10. 7| propose future 
work In ISectlon 10. 8| and conclude In ISectlon 10.51 



10.2 Indexed datatypes 

While the Haskell libraries described In the first part of this thesis already allow a wide 
range of datatypes to be handled In a generic fashion, they cannot deal with Indexed 
datatypes^ We call a datatype indexed If It has a type parameter that Is not used as data 
(also called a phantom type parameter), and at least one of the datatype's constructors 
Introduces type-level constraints on this type. The type of vectors, or size-constrained 
lists, is an example of such a datatype: 

data Vec a n where 

Nil Vec a Ze 

Cons :: a — > Vec a n — > Vec a (Su n) 



The indexed approach of |Chapter 6| can deal with such datatypes, but it remains unclear how to port this 
library to Haskell. 
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The first parameter of Vec, a, Ls the type of the elements of the vector. In the CADT 
syntax above with type signatures for each constructor, we see that a appears as an 
argument to the Cons constructor; a is a regular type parameter. On the other hand, the 
second parameter of Vec, r\, does not appear as a direct argument to any constructor: it 
is only used to constrain the possible ways of building Vecs. We always instantiate rj 
with the following empty (uninhabited) datatypes: 

data Ze 
data Su n 

type T 0 = Ze 
type T, = Su T 0 
type T 2 = Su J, 

A vector with two Chars, for instance, is represented as follows: 

exampleVec :: Vec Char T 2 

exampleVec = Cons 'p' (Cons 'q' Nil) 

Note that its type, Vec Char T 2 , adeguately encodes the length of the vector; giving any 
other type to exampleVec, such as Vec Char T 0 or Vec Char Char, would result in a type 
error. 

Indexed types are easy to define as a GADT, and allow us to give more specific types 
to our functions. For instance, the type of vectors above allows us to avoid the usual 
empty list error when taking the first element of an empty list, since we can define a 
head function that does not accept empty vectors: 

head Vec :: Vec a (Su rf) -* a 
headVec (Cons x _) = x 

GHC correctly recognises that it is not necessary to specify a case for headVec Nil, since 
that is guaranteed never to happen by the type-checker. 

Indexed datatypes are also useful when specifying well-typed embedded languages: 

data Term a where 

Lit :: Int — ► Term Int 

IsZero :: Term Int — » Term Bool 

Pair :: Term a — > Term (3 — » Term (a , f3) 

If :: Term Bool — > Term a — > Term a — > Term a 

The constructors of Term specify the types of the arguments they reguire and the type 
of term they build. We will use the datatypes Vec and Term as representative examples 
of indexed datatypes in the rest of this chapter. 



10.2.1 Type-level equalities and existential quantification 

Indexed datatypes such as Vec and Term can be defined using only existential guantifica- 



tion and type-level egualities Baars and Swierstra 2004. For example, GHC rewrites 



Vec to the following eguivalent datatype: 
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data Vec a q = rj ~ Ze => Nil 

forall li. q ~ Su /.i => Cons a (Vec a /;) 

The constructor Nil introduces the constraint that the type variable q eguals Ze; a ~ /3 
Is GHC's notation for type-level eguallty between types a and /3. The Cons construc- 
tor regulres ;? to be a Su of something; this "something" Is encoded by Introducing 
an exlstentlally-guantlfied variable which stands for the length of the subllst, and 
restricting r\ to be the successor of (in other words, one plus the length of the subllst). 

This encoding of Vec Is entirely egulvalent to the one shown previously. While It may 
seem more complicated, It makes explicit what happens "behind the scenes" when using 
the Nil and Cons constructors: Nil can only be used If ;? can be unified with Ze, and Cons 
Introduces a new type variable constrained to be the predecessor of /?. In the next 
section we will look at how to encode Indexed datatypes generlcally; for this we need to 
know what kind of primitive operations we need to support. Looking at this definition of 
Vec, it Is clear that we will need not only a way of encoding type eguallty constraints, 
but also a way to Introduce new type variables. 

10.2.2 Functions on indexed datatypes 

The extra type safety gained by using Indexed datatypes comes at a price: defining 
functions operating on these types can be harder. Consumer functions are not affected; 
we can easily define an evaluator for Term, for Instance: 

eval :: Term a — > a 

eval (Lit i) = i 

eval (IsZero t) = eval t = 0 

eval (Pair a b) = (eval a , eval b) 

eval (If p a b) = if eval p then eval a else eval b 

In fact, even GHC can automatically derive consumer functions on Indexed datatypes for 
us; the following Show Instances work as expected: 

deriving instance Show a => Show (Vec a rj) 
deriving instance Show (Term a) 

Things get more complicated when we look at producer functions. Let us try to define 
a function to enumerate values. For lists this Is simple: 

enumList :: [a] — * [[a]] 

enumList ea = [] : [x : xs | x <— ea, xs <— enumList ea] 

Given an enumeration of all possible element values, we generate all possible lists, 
starting with the empty listrl However, a similar version for Vec Is rejected by the 
compiler: 

2 We are not diagonalising the elements from ea and enumList ea, but that is an orthogonal issue. 
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enumVec :: [ a] — » [Veccr/7] 

enumVec ea = Nil : [Cons x xs | x <— ea, xs <— enumVec ea] 

GHC complains of being unable to match Ze with Su /?, and rightfully so: we try to add 
Nil, of type Vec a Ze, to a list containing Conses, of type Vec a (Su /?). To make this 
work we can use type classes: 

instance GEnum (Vec a Ze) where 
genum = [Nil] 

instance (GEnum a, GEnum (Vec a /?)) => GEnum (Vec a (Su q)) where 
genum = [Cons at | a <— genum, t <— genum] 

In this way we can provide different types (and implementations) to the enumeration of 
empty and non-empty vectors. 

Note that GHC (version 7.4.1) is not prepared to derive producer code for indexed 
datatypes. Trying to derive an instance Read (Vec a /]) results in the generation of type- 
incorrect code. The difficulties associated with deriving instances for indexed datatypes 
have been known for a while (see, for instance, the GHC bug reports #3012 and #4528|. 
We show in Section 10.4.2 a way of identifying the necessary instances to be defined, 



which could also be used to improve instance deriving for GADTs. 



10.3 Handling Indexing genetically 

As we have seen in the previous section, to handle indexed datatypes generically we 
need support for type egualities and guantification in the generic representation. We 
deal with the former in |Section 1 0.3.1 1 and the latter in |Section 1 0.3.2| 

10.3.1 Type equalities 

A general type eguality a ~ /? can be encoded in a simple GADT: 

data a :=: fj where 

Refl :: a :=: a 

We could add the :=: type to the representation types of instant-generics, and add 
type egualities as extra arguments to constructors. However, since the egualities are 
always introduced at the constructor level, and instant-generics has a representation 
type to encode constructorsj^jwe prefer to define a more general representation type for 
constructors which also introduces a type eguality: 

data CEq y cp ip a where 

CEq :: a — > CEq y 0 0 a 

3 We have elided It from the description in |Chapter 7| but it is present in the distribution of the Library. This 
way of encodiny meta-information is explained in detail inlSection 11.2.21 
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The new CEq type takes two extra parameters which are forced to unify by the CEq 
constructor. The old behavior of C can be recovered by instantiating the 0 and ip 
parameters to trivially equal types: 

type C y a = CEq y () () a 

Note that we can encode multiple equalities as a product of equalities. For example, 
a constructor which introduces the equality constraints a ~ Int and ft ~ Char would be 
encoded with a representation of type CEq y (a :x: ft) (Int :x: Char) 5 (for suitable y 
and 5). 



Encoding types with equality constraints 

At this stage we are ready to encode types with equality constraints that do not rely 
on existential quantification; the :=: type shown before is a good example: 

instance Representable (a :=: ft) where 

type Rep (a :=: ft) = CEq Eq Refl a ft U 

from Refl = CEq U 
to (CEq U) = Refl 

The type equality introduced by the Refl constructor maps directly to the equality in- 
troduced by CEq, and vice-versa. As Refl has no arguments, we encode it with the unit 
representation type U. The auxiliary datatype EqR e n, which we omit, is used to encode 



constructor information about Refl (see Section 11.2.2 for more information on encoding 
meta-information). 



Generic functions over equality constraints 

We need to provide instances for the new CEq representation type for each generic 



function. The instances for the equality and enumeration functions of Section 9.2 



are: 



instance (GEqRep a) => GEqRep (CEq y <fi tp a) where 
geqRep (CEq a) (CEq b) = geqRep a b 

instance (GEnumRep a) => GEnumRep (CEq y 0 (j> a) where 
genumRep = map CEq genumRep 

instance GEnumRep (CEq y <p ip a) where 

genumRep = [ 

Generic consumers, such as equality, are generally not affected by the equality con- 
straints. Generic producers are somewhat trickier, because we are now trying to build 
a generic representation, and thus must take care not to build impossible cases. For 
generic enumeration, we proceed normally in case the types unify, and return the empty 
enumeration in case the types are different. Note that these two instances overlap, but 
remain decidable. 
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10.3.2 Existentially-quantified indices 

Recall the shape of the Cons constructor of the Vec datatype: 

forall \i. q ~ Su /j => Cons a (Vec a /v) 

We need to be able to introduce new type variables in the type representation. A first 
idea would be something like: 

type Rep (Vec a q) — forall ^. CEq Vec Co ns 1 (Su /j) ... 

This however is not accepted by GHC, as the right-hand side of a type family instance 
cannot contain quantifiers. This restriction is well justified, as allowing this would lead 



to higher-order unification problems Neubauer and Thiemann 2002 



Another attempt would be to encode representations as data families instead of type 
families, so that we can use regular existential quantification: 

data instance Rep (Vec a q) = forall \i. RepVec (CEq Vec Co ns 1 (Su . . .) 

However, we do not want to use data families to encode the generic representation, as 
these introduce a new constructor per datatype, thereby effectively precluding a generic 
treatment of all types. 



Faking existentials 

Since the conventional approaches do not work, we turn to some more unconventional ap- 
proaches. All we have is an index type variable /), and we need to generate existentially- 
quantified variables that are constrained by q. We know that we can use type families 
to create new types from existing types, so let us try that. We introduce a type family 

type family X q 

and we will use X // where the original type uses We can now write a generic 
representation for Vec: 

instance Representable (Vec a q) where 
type Rep (Vec a q) — CEq Vec Ni | q Ze U 

:+: CEq Vec Con s 1 (Su (X q)) (Var a :x: Rec (Vec a (X q))) 
from Nil = L (CEq U) 

from (Cons h t) = R (CEq (Var h :x: Rec t)) 

to (L (CEq U)) = Nil 

to (R (CEq (Var h :x: Rec t))) = Cons h t 

This is a good start, but we are not done yet, as GHC refuses to accept the code above 
with the following error: 

4 This is closely related to Skolemization. 
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Could not deduce (m ~ X (Su m)) 

from the context (n ~ Su m) 

bound by a pattern with constructor 

ConsVec :: forall an. a -> Vec a n -> Vec a (Su n) , 
in an equation for 'from' 

What does this mean? GHC is trying to unify ^ with X (Su fj), when it only knows that 
q ~ Su jj. The eguality q ~ Su fj comes from the pattern-match on Cons, but why is 
it trying to unify with X (Su fj)l Well, on the right-hand side we use CEq with type 
CEq Veccons 1 (Su (X q)) . . ., so GHC tries to prove the eguality q ~ Su (X q). In trying 
to do so, it replaces q by Su \i, which leaves Su /v ~ Su (X (Su /v)), which is implied by 
IJ ~ X (Su y), but GHC cannot find a proof of the latter eguality. 

This is unsurprising, since indeed there is no such proof. Fortunately we can supply 
it by giving an appropriate type instance: 

type instance X (Su /j) = /j 

We call instances such as the one above "mobility rules", as they allow the index to 
"move" through indexing type constructors (such as Su) and X. Adding the type instance 
above makes the Representable instance for Vec compile correctly. Note also how X 
behaves much like an extraction function, getting the parameter of Su. 



Representation for Term. The Term datatype (shown in Section 10.2) can be rep- 
resented generically using the same technigue. First let us write Term with explicit 
guantification and type egualities: 

data Term a = a ~ Int => Lit Int 

a ~ Bool => IsZero (Term Int) 
forall /? y. a ~ (/? , y) => Pair (Term /?) (Term y) 

If (Term Bool) (Term a) (Term a) 

We see that the Lit and IsZero constructors introduce type egualities, and the Pair con- 
structor abstracts from two variables. This means we need two type families: 

type family X-^ a 
type family X 2 a 

Since this strategy could reguire introducing potentially many type families, we use a 
single type family instead, parameterised over two other arguments: 

type family X y t a 

We instantiate the y parameter to the constructor representation type, i to a type-level 
natural indicating the index of the introduced variable, and a to the datatype index 
itself. 

The representation for Term becomes: 
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type Repierm a = CEq Term Lit a Int 
(Rec Int) 

:+: CEq Termi sZero a Bool 
(Rec (Term Int)) 

:+: CEq Term Pair a (X Term Pair T 0 a , X Term Pair 1^ a) 
( Rec (Term (X Term Pair T 0 a)) 
:x: Rec (Term (X Term Pair a))) 

:+: C Termif 

(Rec (Term Bool) :x: Rec (Term a) :x: Rec (Term a)) 

We show only the representation type Repierm, as the from and to functions are trivial. 
The mobility rules are induced by the equality constraint of the Pair constructor: 

type instance X Term Pair T 0 (/? , y) = (3 
type instance X Term Pair ()8 , y) = y 

Again, the rules resemble selection functions, extracting the first and second components 
of the pair. 

Summarising, quantified variables are represented as type families, and type equal- 
ities are encoded directly in the new CEq representation type. Type equalities on 
quantified variables need mobility rules, represented by type instances. We have seen 
this based on two example datatypes; in Section 10.5 we describe more formally how to 



encode indexed datatypes in the general case. 



10.4 Instantiating generic functions 

Now that we know how to represent indexed datatypes, we proceed to instantiate generic 
functions on these types. We split the discussion into generic consumers and producers, 
as they require a different approach. 



10.4.1 Generic consumers 

Instantiating generic equality to the Vec and Term types is unsurprising: 

instance (GEq a) => GEq (Vec a q) where 
geq = geq Default 

instance GEq (Term a) where 

geq = geq Default 



Using the instance for generic equality on CEq of Section 1 0.3.1 1 these instances compile 



and work fine. The instantiation of generic consumers on indexed datatypes is therefore 
no more complex than on standard datatypes. 
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10.4.2 Generic producers 



Instantiating generic producers is more challenging, as we have seen in Section 10.2.2 
For Vec, a first attempt could be: 

instance (GEnum a) => GEnum (Vec a q) where 
genum = genumDefault 

However, this will always return the empty list: GHC's instance resolution mechanism 
will look for a GEnumRep instance for the heads CEq y Ze i] ft and CEq y (Su f/) i] ft (for 
some y and ft which are not relevant to this example). In both cases, only the second 
GEnumRep instance of Section 1 0.3.1 1 applies, as neither Ze nor Su fj can be unified 



with /?. 

Therefore, and as before, we need to give two instances, one for Vec a Ze, and another 
for Vec a (Su /?), given an instance for Vec a q: 

instance (GEnum a) => GEnum (Vec a Ze) where 
genum = genumDefault 

instance (GEnum a, GEnum (Vec a q)) => GEnum (Vec a (Su q)) where 
genum = genumDefault 

We can check that this works as expected by enumerating all the vectors of Booleans of 
length one: genum :: [Vec Bool (Su Ze)] evaluates to [Cons True Nil, Cons False Nil], the 
two possible combinations. 



Instantiating Term. Instantiating GEnum for the Term datatype follows a similar strat- 
egy. We must identify the types that Term is indexed on. These are Int, Bool, and (a , ft), 
in the Lit, IsZero, and Pair constructors, respectively. The If constructor does not impose 
any constraints on the index, and as such can be ignored for this purpose. Having 
identified the possible types for the index, we give an instance for each of these cases: 

instance GEnum (Term Int) where 
genum = genumDefault 

instance GEnum (Term Bool) where 
genum = genumDefault 

instance (GEnum (Term a), GEnum (Term ft)) => GEnum (Term (a , ft)) where 
genum = genumDefault 

We can now enumerate arbitrary Terms. However, having to write the three instances 
above manually is still a repetitive and error-prone task; while the method is trivial 
(simply calling genumDefault), the instance head and context still have to be given, but 
these are determined entirely by the shape of the datatype. We have written Template 
Haskell code to automatically generate these instances for the user. 

In this section and the previous we have seen how to encode and instantiate generic 
functions for indexed datatypes. In the next section we look at how we automate this 
process, by analysing representation and instantiation in the general case. 
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In general, an Indexed datatype has the following shape: 
data Da = V/?7.yT => 07 

| V£„.y^ => C n <f> n 

We consider a datatype D with arguments a (which may or may not be Indices), and n 
constructors Ci ... C n , with each C constructor potentially Introducing exlstentlally- 
guantlfied variables /?,-, type egualltles y], and a list of arguments cf>j. We use an overllne 
to denote seguences of elements. 

We need to Impose some further restrictions to the types we are able to handle: 

1. Quantified variables are not allowed to appear as standalone arguments to the 
constructor: ^\p e j r fi £ </>;• 

2. Quantified variables have to appear In the eguallty constraints: V; ^ E -^.Btp.ip j?6 y[. 
We regulre this to provide the mobility rules. 

For such a datatype, we need to generate two types of code: 

1. The generic representation 

2. The Instances for generic Instantiation 

We deal with (1) ln|Sectlon 1051] and (2) ln|Sectlon 1052 



10.5.1 Generic representation 

Most of the code for generating the representation Is not specific to Indexed datatypes; 



see Section 11.4 for a definition of a similar representation. The code generation for 
constructors needs to be adapted from the standard definition, since now CEq takes two 
extra type arguments. The value generation (functions from and to) Is not affected, only 
the representation type. 



Type equalities. For each constructor C an eguallty constraint Ti ~ T2 S Yi becomes 
the second and third arguments to CEq, for Instance CEq ... Ti tj .... Multiple con- 
straints like Ti ~ T2, T3 ~ T4 become a product, as In CEq ... (n :x: T3) (T2 :x: T4) . . .. 
An exlstentlally-quantlfled variable f3, appearing on the right-hand side of a constraint 
of the form t ~ ... or on the arguments to C Is replaced by X y l t, with y the 
constructor representation type of C, and 1 a type-level natural version of i. 
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MobULtii rules. For the generated type families we provide the necessary mobility rules 
Section 10.3.2 . Given a constraint V/3„.y^, each eguality f3 ~ ijj t, where ft S /? and 
ip t is some type expression containing t, we generate a type instance X y t (<// t) = t, 
where y is the constructor representation type of the constructor where the constraint 
appears, and t is a type-level natural encoding the index of the constraint. As an 
example, for the Cons constructor of Section 10.2.1 f3 is /?, r is ip is Su, y is Veccons. 
and ( is T 0 (since there is only one constraint). 



10.5.2 Generic instantiation 

To give a more thorough account of the algorithm for generating instances we sketch its 
implementation in Haskell. We assume the following representation of datatypes: 

data Datatype = Datatype [ TyVar ] [ Con ] 
data Con = Con Constraints [ Type ] 

data Type — abstract 
data TyVar — abstract 

tyVarType :: TyVar -* Type 

A datatype has a list of type variables as arguments, and a list of constructors. Con- 
structors consist of constraints and a list of arguments. For our purposes, the particular 
representation of types and type variables is not important, but we need a way to convert 
type variables into types (tyVarType). 

Constraints are a list of (existentially-guantified) type variables and a list of type 
egualities: 

data Constraints = [TyVar] > [TyEq] 
data TyEq = Type :~: Type 

We will need eguality on type egualities, so we assume some standard eguality on types 
and type egualities. 

We start by separating the datatype arguments into "normal" arguments and indices: 
findlndices :: Datatype — > [TyVar :+: TyVar] 

findlndices (Datatype vs cs) = [if v 'inArgs' cs then L v else R v | v <— vs] 
inArgs :: TyVar — » [Con] — > Bool 
inArgs = . . . 

We leave inArgs abstract, but its definition is straightforward: it checks if the argument 
TyVar appears in any of the constructors as an argument. In this way, findlndices tags 
normal arguments with L and potential indices with R. These are potential indices 
because they could also just be phantom types, which are not only not used as argument 
but also have no equality constraints. In any case, it is safe to treat them as indices. 
The islndex function used before is defined in terms of findlndices: 
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islndex :: TyVar — > Datatype — > Bool 
islndex t d = R t £ findlndices d 

Having Identified the Indices, we want to Identify all the return types of the construc- 
tors, as these correspond to the heads of the instances we need to generate. This Is the 
task of function findRTs: 

findRTs :: [TyVar] — > [Con] — > [Constraints] 

findRTs is [] = [] 

findRTs is ((Con cts args) : cs) = let rs = findRTs is cs 

in if anyln is cts then cts : rs else rs 

anyln :: [TyVar] — > Constraints — > Bool 

anyln vs (_> teqs) = or [v 'in TyEq ' teqs | v <- vs] 

iriTyEq :: TyVar — > [TyEq] — > Bool 

i n TyEq = • • • 

We check the constraints In each constructor for the presence of a type equality of the 
form i :~: t, for some Index type variable i and some type t. We rely on the fact that 
GADTs are converted to type equalities of this shape; otherwise we should look for the 
symmetric equality t :~: i too. 

Having collected the Important constraints from the constructors, we want to merge 
the constraints of the constructors with the same return type. Given the presence of 
quantified variables, this Is not a simple equality test; we consider two constraints to 
be equal modulo all possible Instantiations of the quantified variables: 

instance Eq Constraints where 

(vs > cs) = (ws > ds) = length vs = length ws A cs = subst ws vs ds 

subst :: [TyVar] -> [TyVar] -> [TyEq] — > [TyEq] 
subst vs ws teqs = . . . 

Two constraints are equal If they abstract over the same number of variables and their 
type equalities are the same, when the quantified variables of one of the constraints 
are replaced by the quantified variables of the other constraint. This replacement Is 
performed by subst; we do not show Its code since It Is trivial (given a suitable definition 
of Type). 

Merging constraints relies on constraint equality. Each constraint Is compared to 
every element In an already merged list of constraints, and merged If It Is equal: 

merge :: Constraints — > [Constraints] — > [Constraints] 
merge ci [] = [ci] 

merge Ci@(vs > cs) (C2@(ws > ds) : ess) 
| Ci = C2 = Ci : ess 
| otherwise = c 2 : merge Ci ess 
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mergeConstraints :: [Constraints] — > [Constraints] 
mergeConstraints = foldr merge [] 

We can now combine the functions above to collect all the merged constraints: 

rightOnly :: [a :+: [£] 
rightOnly I = [x | R x <— I] 

allConstraints :: Datatype — > [Constraints] 

allConstraints d@(Datatype _ cons) = let is = rightOnly (findlndices d) 

in mergeConstraints (findRTs is cons) 

We know these constraints are of shape i :~: t, where i is an index and t is some type. We 
need to generate instance heads of the form instance G (D a), where a S buildlnsts D. 
The function buildlnsts computes a list of type variable instantiations starting with the list 
of datatype arguments, and instantiating them as dictated by the collected constraints: 

buildlnsts :: Datatype — > [[Type]] 
buildlnsts d@(Datatype ts _) = map (instVar ts) cs 
where cs = concat (map (A (_ > t) -* t) (allConstraints d)) 

instVar :: [TyVar] — ► TyEq — > [Type] 
instVar[] _ = [] 
instVar (v : vs) tEq@(i :~: t) 

| tyVarType v = i = t : map tyVarType vs 

| otherwise = tyVarType v : instVar vs tEq 

This completes our algorithm for generic instantiation to indexed datatypes. As men- 
tioned before, the same analysis could be used to find out what Read instances are 
necessary for a given indexed datatype. Note however that this algorithm only finds 
the instance head for an instance; the method definition is trivial when it is a generic 
function. 



10.6 Indices of custom kinds 



So far we have only considered datatypes with indices of kind *. However, a recent GHC 
extension allows programmers to define their own kinds, paving the way for GADTs that 
are indexed over indices of kind other than * Yorgey et al. 2012. Custom indices 



allow for more type (or kind) safety. The type of vectors introduced in Section 10.2 for 
instance, can be given a more precise definition using data kinds: 

data Nat = Ze | Su Nat 

data Vec a :: Nat —* * where 

Nil Vec a Ze 

Cons :: a — > Vec a q — > Vec a (Su rj) 
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Note the subtle difference: previously we defined two empty datatypes Ze and Su /?, of 
kind *, and used those as indices of Vec. With the -XDataKinds language extension, 
we can instead define a regular Nat datatype for natural numbers, with constructors Ze 
and Su. By means of promotion, the Nat declaration also introduces a new kind Nat, 
inhabited by two types, Ze and Su[^] We can then give a more detailed kind to Vec, 
restricting its second argument (an index) to be of the newly-defined kind Nat. The 
return type of the Nil constructor, for instance, is Vec a Ze, where Ze has kind Nat, and 
not * as in ISection 10.21 

An immediate advantage of using datakinds for indices is that it reduces the potential 
for errors. The type Vec Int Bool, for instance, now gives rise to a kind error, since Bool 



has kind *, and not Nat. The Term datatype of Section 10.2 can also be encoded with 
datakinds: 

data Term, = Inti | Booli | Pair, Term, Term, 

data Term :: Term, — > * where 

Lit :: Int — ► Term Inti 

IsZero :: Term Inti — * Term Booli 
Pair :: Term a — > Term (3 — > Term (Pain a f3) 

If :: Term Bool, — » Term a — > Term a —*■ Term a 

We define a new datatype Ternij standing for the type of indices of Term. We are not 
really interested in the values of Ternij, only in its promotion; we use the promoted kind 
Term, in the kind of Term, and the promoted constructors in the types of the constructors 
of Term. 

The mechanism to deal with indexed datatypes that we have explained so far focuses 
on indices of kind *. Fortunately, with one small change, it can egually well deal with 
other kinds. Recall the kind of the X representation type from Section 10.3.2 



type family X (y :: *) (( :: *) (a :: *) :: * 

It states that the kind of datatype indices, a, is *. This is too restrictive if we want to 
use indices of kind Nat or Ternij. We could define a new type family for each datatype 
to represent, with the appropriate kind for the a parameter, but this is repetitive and 
obscures the fact that X should be independent of the datatype being represented. 
Fortunately, there is also a GHC extension for kind polymorphism, which is exactly 
what we need for defining X. Its kind becomes: 

type family X (y :: ★) (t :: Nat) (a :: k) :: k 

We abstract from the kind of indices with a kind variable k. Since X will replace an 
index, its kind has to be the same as that of the index, so the return kind of X is k. Note 



5 Since types and constructors can have the same name in Haskell, there might be ambiguity when resolving 
a potentially promoted name. In source code, promoted constructors can be prefixed with a single quote 
as a means to resolve the ambiguity. In our presentation, we use colours to distinguish between syntactic 
categories. 
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also that we can improve the kind of (, the natural number used to distinguish between 
multiple indexed variables. 

Changing the kind of X is all that it takes to support datakinds as indices with 
our approach; the remainder of the description, including the instantiation of generic 
functions, remains unchanged. 



1 0.7 Related work 



Indexed datatypes can be seen as a subset of all GADTs, or as existentially-guantified 
datatypes using type-level egualities. Johann and Ghani 2008 developed a categorical 
semantics of GADTs, including an initial algebra semantics. While this allows for a 
better understanding of GADTs from a generic perspective, it does not translate directly 
to an intuitive and easy-to-use generic library. 



Gibbons 2008 describes how to view abstract datatypes as existentially-guantified, 



and uses final coalgebra semantics to reason about such types. Rodriguez Yakushev and 



Jeuring |201 0 describe an extension to the spine view Hinze et a I. 2006 supporting ex 



istential datatypes. Both approaches focus on existentially-guantified data, whereas we 
do not consider this case at all, instead focusing on (potentially existentially-guantified) 
indices. See lSection 1 0.8| for a further discussion on this issue. 

Within dependently-typed programming, indexing is an ordinary language feature 
which can be handled generically more easily due to the presence of type-level lambdas 



and explicit type application, as we have seen in Section 6.1.8 



1 0.8 Future work 

While we can express indexed datatypes both as GADTs and as existentially-guantified 
datatypes with type-level egualities, the reverse is not true in general. Consider the 
type of dynamic values: 

data Dynamic = forall a. Typeable a => Dyn a 

The constructor Dyn stores a value on which we can use the operations of type class 
Typeable, which is all we know about this value. In particular, its type is not visible 
"outside", since Dynamic has no type variables. Another example is the following variation 
of Term: 



data Term a where 

Const :: a — > Term a 

Pair :: Term a — > Term — > Term (a , (3) 

Fst :: Term (a , f3) — > Term a 

Snd :: Term (a , f3) —* Term )8 

Here, the type argument a is not only an index but it is also used as data, since values of 
its type appear in the Const constructor. Our approach cannot currently deal with such 
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datatypes. We plan to Investigate if we can build upon the work of |Rodrlguez Yakushev 



and Jeurlng 2010 to also support exlstentlals when used as data. 



10.9 Conclusion 

In this chapter we have seen how to Increase the expressiveness of a generic programming 
library by adding support for Indexed datatypes. We used the instant-generics 
library for demonstrative purposes, but we believe the technlgue readily generalises 
to all other generic programming libraries using type-level generic representation and 
type classes. We have seen how Indexing can be reduced to type-level egualltles and 
existential guantlflcatlon. The former are easily encoded In the generic representation, 
and the latter can be handled by encoding the restrictions on the guantlfled variables 
as relations to the datatype Index. All together, our work brings the convenience and 
practicality of datatype-generic programming to the world of Indexed datatypes, widely 
used In many applications but so far mostly Ignored by generic programming approaches. 
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CHAPTER 1 1 



Integrating generic programming in Haskell 



Haskell's deriving mechanism supports the automatic generation of Instances for a num- 
ber of functions. The Haskell 98 Report Peyton Jones 2003 only specifies how to 
generate Instances for the Eq, Ord, Enum, Bounded, Show, and Read classes. The de- 
scription of how to generate Instances Is largely Informal, and It Imposes restrictions on 
the shape of datatypes, depending on the particular class to derive. As a consequence, 
the portability of Instances across different compilers Is not guaranteed. 

In this chapter we describe a new approach to Haskell's deriving mechanism In which 
Instance derivation Is realised through default methods. This allows users to define 
classes whose Instances require only a single line of code specifying the class and 
the datatype, similarly to the way deriving currently works. We use this to leverage 
generic programming In Haskell; generic functions, Including the methods from the above 
six Haskell 98 derivable classes, can then be specified In Haskell Itself, making them 
lightweight and portable. 



11.1 Introduction 



Generic programming has come a long way: from Its roots In category theory, passing 
through dedicated languages, language extensions and pre-processors until the flurry 
of library-based approaches of today {Section 8.1) . In this evolution, expressivity has 
not always Increased: many generic programming libraries of today still cannot com- 
pete with Generic Haskell, for Instance. The same applies to performance, as libraries 
tend to do little regarding code optimisation, whereas meta-programmlng techniques 
such as Template Haskell |Sheard and Peyton Jones" 2002 can generate near-optimal 
code. Instead, generic programming techniques seem to evolve In the direction of better 
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availability and usability: it should be easy to define generic functions and it should 
be trivial to use them. Certainly some of the success of the SYB approach is due to its 
availability: it comes with GHC, the main Haskell compiler, which can even derive the 
necessary type class instances to make everything work with no effort. 

To improve the usability of generics in Haskell, we believe a tighter integration with 
the compiler is necessary. In fact, the Haskell 98 standard already contains some 



generic programming, in the form of derived instances Peyton Jones 2003 Chapter 
10]. Unfortunately, the report does not formally specify how to derive instances, and it 
restricts the classes that can be derived to six only (Eq, Ord, Enum, Bounded, Show, and 
Read). GHC has since long extended these with Data and Typeable (the basis of SYB), 
and more recently with Functor, Foldable, and Traversable. Due to the lack of a unifying 
formalism, these extensions are not easily mimicked in other compilers, which need to 
reimplement the instance code generation mechanism. 
To address these issues we: 

• Design a new generic programming library for Haskell. Our library borrows heavily 
from other approaches to generic programming in Haskell, but pays particular 
attention to practical usage concerns. In particular, it can represent the most 
common forms of datatypes, and encode the most commonly used generic functions, 
while remaining practical to use. 

• Show how this library can be used to replace the deriving mechanism in Haskell, 
and provide some examples, such as the Functor class. 

• Provide a detailed description of how the representation for a datatype is gener- 
ated. In particular, we can represent almost all Haskell 98 datatypes. 



Implemented our approach in both the Utrecht Haskell Compiler [UHC, Dijkstra 



et a I. 2009 , and in GHC. The implementation in UHC reflects an earlier version of 
our proposal, and makes use of very few language extensions. The implementation 
in GHC uses type families in a way similar to instant-generics {Section 7.2) . 
We also provide a package that shows in detail the code that needs to be added 
to the compiler, the code that should be generated by the compiler, and the code 
that is portable between compilers^] 



This chapter is an updated version of previous work Magalhaes et a I. 2010a. We 
describe the design as it is implemented in GHC|^] 

We continue this chapter by introducing the generic programming library designed for 
our extension {Section 1 1 ,2| , We proceed to show how to define generic functions {Sec- 



tion 1 1 .3] , and then describe the necessary modifications to the compiler for supporting 



our approach {Section 11. 4| . In Section 11.5 we explain the differences between the 



current description and our earlier design, which is implemented in UHC. Finally, we 



1 http : //hackage . haskell . org/ package/generi c-derivingl 

bee also the description on the UHC user's guide: |http://www.haskell.or g/ghc/do cs/7.4. l/html/| 
user s_guide/gener ic-programming . html 
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discuss alternative designs {Section 11,6) , review related work {Section 11.7) , propose 
future work {Section 1 1 .8) , and conclude in Section 11.9 



1 1 .2 The generic-deriving library 

We begin with an example of what we want generic programming usage to look like 
in Haskell. We use the generic function encode as a running example throughout this 
chapter. This function transforms a value into a seguence of bits: 

data Bit = O | I 

class Encode a where 
encode :: a — > [Bit] 

A user should be able to define a datatype and ask for a generic implementation of 
encode, similarly to what happens for show: 

data Exp = Const Int | Plus Exp Exp 
deriving (Show, Encode) 

However, in our proposal we do not use deriving for generating instances. Instead, we 
propose to use an empty instance head: 

instance Encode Exp 

Using a special form of default method declaration in the Encode class (which we discuss 
in detail in |Section 11.3.3 , this empty instance definition gives rise to an instance of 



a generic implementation of encode for the type Exp. The function is then ready to be 
used: 

test :: [Bit] 

test = encode (Plus (Const 1) (Const 2)) 

This should be all that is necessary to use encode. The user should need no further 
knowledge of generics, and encode can be used in the same way as show, for instance. 

Behind the scenes, the compiler uses a generated representation of Exp to instantiate 
a generic definition of encode. For this we reguire a generic representation, which we 
now describe. We discuss alternative designs and motivate our choice in more detail 
Section 1 1 .6 The new library, which we call generic-deriving, is a mix between 



in 



polyp {Chapter 4) and instant-generics Chapter"?) , in the sense that it supports 



one datatype parameter, but it does not have a fixed-point view on data. 
11.2.1 Run-time type representation 

The choice of a run-time type representation affects not only the compiler writer, who has 
to implement the automatic generation of representations, but also the expressiveness of 
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the whole approach. A simple representation is easier to derive, but might not allow for 
defining certain generic functions. More complex representations are more expressive, 
but reguire more work for the automatic derivation of instances. 

We present a set of representation types that tries to balance these factors. We use 
the common sum-of-products representation without explicit fixpoints but with explicit 
abstraction over a single parameter. Therefore, representable types are functors, and we 
can compose types. Additionally, we provide useful types for encoding meta-information 
(such as constructor names) and tagging arguments to constructors. We show examples 



of how these representation types are used in Section 11.2.4 



The basic ingredients of our representation are units, sums, and products: 



data li-i p = Ui 

data (:+:) <p p p = U {unl^ :: <p p} | {unF^ :: pi p} 
data (:x:) <p ip p = cj> p :x: p p 

Like in the implementation of polyp {Section 4.2| , we use a type parameter p to encode 
one datatype parameter. We also have lifted void (V-i) to represent nullary sums, but 
for simplicity we omit it from this discussion and from the generic functions in the next 
section. 

We use an explicit combinator to mark the occurrence of the parameter: 



newtype Pan p = Pan {unPan :: p} 



As our representation is functorial, we can encode composition. We reguire the first 



argument of composition to be a representable type constructor (see Section 11.4 for 
more details). The second argument can only be the parameter, a recursive occurrence 
of a functorial datatype, or again a composition. We use Rec^ to represent recursion, 
and :o: for composition: 

newtype Rec^ p> p — Rec^ {unRec^ :: p p} 
newtype (:o:) <p p p = Comp! (0 (ip p)) 

This is similar to the way polyp treats composition, but now without explicit fixed points. 
Finally, we have two types for tagging and representing meta-information: 



newtype ly p = {unK^ :: y} 
newtype iy p> p = {unl^ :: p p) 

We use K-\ for tagging and M-\ for storing meta-information. The i parameter is used for 
grouping different types of meta-information together, as can be seen in the following 
type synonyms: 
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data D 
data C 
data S 

data R 
data P 



type D, = Mi D 

type d = Mi C 

type S! = S 

type Rec 0 = Kt R 

type Par 0 = P 



We use Reco to tag occurrences of (possibly recursive) types of kind * and Par 0 to mark 
additional parameters of kind *. For meta-information, we use D-i for datatype informa- 
tion, Ci for constructor information, and Si for record selector information. We group 
five combinators into two because in many generic functions the behavior is independent 
of the meta-information or tags. In this way, fewer trivial cases have to be given. For 
instance, generic map {Section 11.3.4| h as a single instance for the Mi representation 
type, while generic show {Section 11.3.6| has separate instances for D-|, and S-|. We 
present the meta-information associated with Mi in detail in the next section. 

Note that we abstract over a single parameter p of kind *. This means we will be 
able to express generic functions such as 

fmap :: (a — > )3) — > <p a — > <p )3 



bimap :: (a -* y) -* (/? -* 5) ->■ <p a /? -* <j) y 5 

For bimap we would need a type representation that can distinguish between the pa- 
rameters. All representation types would need to carry one additional type argument. 
However, in practice few generic functions reguire abstraction over more than a single 
type parameter, so we support abstraction over one parameter only. 

11.2.2 Meta-information 

Some generic functions need information about datatypes, constructors, and records. 
This information is stored in the type representation: 

class Datatype y where 

datatypeName :: a y (cj) :: * — > *) p — > String 
moduleName :: a y (</>::* — > *) p — > String 

class Constructor y where 

conName :: a y {<p :: * — > *) p — > String 

conFixity :: a y (cf> :: * — > *) p — > Fixity 
conFixity = const Prefix 

conlsRecord :: a y (<£::*—►*) p — > Bool 
conlsRecord = const False 



but not 
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class Selector y where 

selName :: <r y (</>::*—►*) p String 

Names are ungualified. We provide the datatype name together with the name of the 
module in which it is defined. This is the only meta-information we store for a datatype, 
although it could be easily extended to add the kind, for example. We only store the 
name of a selector. For a constructor, we also store its fixity and record if it has fields. 
This last information is not strictly necessary, as it can be inferred by looking for non- 
empty selNames, but it simplifies some generic function definitions. The datatypes Fixity 
and Associativity are unsurprising: 

data Fixity = Prefix | Infix Associativity Int 

data Associativity = LeftAssociative | RightAssociative | NotAssociative 

We provide default definitions for conFixity and conlsRecord to simplify instantiation for 
prefix constructors that do not use record notation]^] 

The types of the meta-information functions are defined so that they match the type of 
the representation types. We provide more details in |Section 1 1 .4.5 and the examples 



later in Section 11.2.4 and |Section 1 1 .3.6 also clarify how these classes are used 



Note that we could encode the meta information as an extra argument to M 

data M-t l 4> p = Meta (0 p) 
data Meta = Meta String Fixity . . . 

However, with this encoding it is harder to write generic producers, since to produce 
an M-i we have to produce a Meta for which we have no information. With the above 
representation we avoid this problem by using type-classes to fill in the right information 
for us. Section 1 1 .3.5 shows an example of how this works. 



11.2.3 A generic view on data 

We obtain a generic view on data by defining an embedding-projection pair between a 
datatype and its type representation. We use the following classes for this purpose: 

class Generic a where 
type Rep a :: * — > * 

from :: a — > Rep a x 
to :: Rep a x —* a 

class Generici <j> where 
type Rep! cf> :: * -* * 

3 Note that all constructor arguments will be wrapped In an Si, Independently of using record notation or 
not. We omit this In the example representations of this section for space reasons, but It becomes clear In 
[Section 1"T4] 
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from! :: cf) p — > Rep! (f> p 
toi :: Repi (p P —* <t> P 

We use a type family to encode the representation of a standard type. In Generic! 
we encode types of kind * — > *, so we have the parameter p. In Generic there is no 
parameter, so we invent a variable / which is never used. 

All types need to have an instance of Generic. Types of kind * — > * also need an 
instance of Generic . This separation is necessary because some generic functions (like 
fmap or traverse) require explicit abstraction from a single type parameter, whereas others 
(like show or enum) do not. Given the different kinds involved, it is unavoidable to have 
two type classes for this representation. Note, however, that we have a single set of 
representation types (apart from the duplication for tagging recursion and parameters). 



11.2.4 Example representations 

We now show how to represent some standard datatypes. Note that all the code in this 
section is automatically generated by the compiler, as described in Section 11.4 



Representing Exp. The meta-information for datatype Exp (defined in the beginning 
of this section) looks as follows: 

data $Exp 
data $Const Exp 
data $Plus Exp 

instance Datatype $Exp where 

moduleName _ = "ModuleName" 
datatypeName _ = "Exp" 

instance Constructor $Const Exp where conName _ = "Const" 
instance Constructor $PIusex P where conName _ = "Plus" 

In moduleName, "ModuleName" is the name of the module where Exp lives. The particular 
datatypes we use for representing the meta-information at the type-level are not needed 
for defining generic functions, so they are not visible to the user. In this description we 
prefix them with a $. 

The type representation ties the meta-information to the sum-of-products represen- 
tation of Exp: 

type Repo Xp = $Exp ( d $Const Exp (Rec 0 Int) 

:+: d $Plus Exp (Rec 0 Exp :x: Rec 0 Exp)) 



Note that the representation is shallow: at the recursive occurrences we use Exp, and 
not Rep 0 xp . 

The embedding-projection pair implements the isomorphism between Exp and Repo* p : 
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instance Generic Exp where 
type Rep Exp = Repo* p 

from (Const n) = Mi (U (Mi (Ki n))) 

from (Plus e e') = (Ri (M, (K, e :x: e'))) 

to (Mi (U (Mi (Ki n)))) = Const n 

to (Mi (Ri (Mi (Ki e :x: e')))) = Plus e e' 

Here it is clear that from and to are inverses: the pattern of from is the same as the 
expression in to, and vice-versa. 

Representing lists. The representation for a type of kind *—»•-* reguires an instance 
for both Generic! and Generic. For the standard Haskell list type [] we generate the 
following code: 

type Rep^' p = $List ( d $Nil List 

:+: d $Cons Lis t (Par 0 p :x: Rec 0 [p])) 

instance Generic [ p } where 
type Rep [p] = Repg ist p 

from [] = Mi (U (Mi Ui)) 

from (h : t) = M, (R, (Mi (Ki h :x: t))) 

to (Mi (L, (Mi Ui))) = 0 

to (Mi (Ri (Mi (Ki h :x: K, t)))) = h : t 

We omit the definitions for the meta-information, which are similar to the previous 
example. We use Par 0 to tag the parameter p, as we view lists as a kind * datatype for 
Generic. This is different in the Generici instance: 

type Rep^ ist = D t $List ( d $Nil List Ui 

:+: Ci $Cons Li st (Par! :x: Red [])) 

instance Generic! [] where 
type Rep! [] = Rep\ ist 

from! [] = Mi (Li (Mi U^) 
from! (h : t) = Mi (Ri (Mi (Pan h :x: Reci t))) 
t 0l (Mi (Li (Mi Hi))) = 0 

loi (Mi (Ri (Mi (Pan h :x: Red t)))) = h : t 

We treat parameters and recursion differently in Repo' st and Rep! ist . In Repg st we use 
Par 0 and Reco for mere tagging; in Rep^' we use Par! and Reci instead, which store the 
parameter and the recursive occurrence of a type constructor, respectively. We will see 
later when defining generic functions {Section 1 1 ,3| how these are used. 

Note that while there is still some code duplication, because there are two instances 
for generic representations, we do not have to duplicate the entire set of representation 
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types, since we can represent a datatype without parameters with our lifted represen- 
tation. 

Representing type composition. We now present a larger example, Involving more com- 
plex datatypes, to show the expressiveness of our approach. Datatype Expr represents 
abstract syntax trees of a small language: 

infixr 6 :*: 

data Expr p = Const Int 

Expr p :*: Expr p 
EVar {unVar :: Var p} 
Let [ Decl p ] (Expr p) 

data Decl p = Decl (Var p) (Expr p) 

data Var p = Var p | VarL (Var [ p ]) 

Note that Expr makes use of an Infix constructor (:*:), has a selector (unVar), and uses 
lists In Let. Datatype Var Is nested, since In the VarL constructor Var Is called with [p]. 
These oddities are present only for Illustrating how our approach represents them. We 
show only the essentials of the encoding of this set of mutually recursive datatypes, 
starting with the meta-lnformatlon: 

data $:*: Expr 
data $EVar Expr 

data $ UnVar 

instance Constructor $:*: Expr where 
conName _ = " :*: " 
conFixity _ = Infix RightAssociative 6 

instance Constructor $EVar Expr where 
conName _ = "EVar" 
conlsRecord _ = True 

instance Selector $UnVar where 
selName _ = "unVar" 

We have to store the fixity of the :*: constructor, and also the fact that EVar has a 
record. We store Its name In the Instance for Selector, and tie the meta-lnformatlon to 
the representation: 

type Repf xpr = $Expr 

( ( Ci $Const Exp r (Rec 0 Int) 

:+: C-i $:*: Expr (Red Expr :x: Red Expr)) 
:+: ( C t $EVar Expr (S, $UnVar (Red Var)) 

:+: C, $Let Expr (([] :°: Red Decl) :x: Red Expr))) 
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In Repf xpr we see the use of Si- Also interesting is the representation of the Let con- 
structor: the list datatype is applied not to the parameter p but to Decl p, so we use 
composition to denote this. Note also that we are using a balanced encoding for the 
sums (and also for the products). This improves the performance of the type-checker, 
and makes generic encoding more space-efficient. 

We omit the representation for Decl. For Var we use composition again: 

type Repf r = $Var ( d $Var Var Par! 

:+: d $VarL Var (Var :o: Red [])) 

In the representation of the VarL constructor, Var is applied to [p\. We represent this as 
a composition with Reci []. 

When we use composition, the embedding-projection pairs become slightly more com- 
plicated: 

instance Generic! Expr where 



type Rep! Expr 



Expr 



from! 
from! 
fromi 
fromi 

to! (M 
td (M 
to! (M 
td (M 



Repr 

Const i) = Mi (Li (Li (M, (K, I)))) 
ei :*: e 2 ) = M-i (L-\ (R-i (M-i (Red ei :x: Red e 2 )))) 
EVar v) = M, (R, (L, (M, (l^ (Red v))))) 
Let d e) = Iv^ (R-\ (R-t (M-\ (Comp! (gmap Red d) 

(U (Li (M! (Ki I))))) = Const i 

(U (Ri (Mi (Red e-t :x: Red e 2 ))))) = e\ :*: e 2 

(Ri (Li (Mi (M, (Red v)))))) = EVarv 

(R-i (R-\ (M-i (Comp! d :x: Rec! e))))) = Let (gmap unRec! d) e 



Red e)))) 



We need to use gmap to apply the Red constructor inside the lists. In this case we could 
use map instead, but in general we reguire the first argument of :o: to have a GFunctor 
instance so we can use gmap (see Section 11. 3| . In toi we need to convert back, this 
time mapping unReCi. Note that the cases for EVar use two seguential M-t constructors, 
one for constructor and the other for record selector meta-information. 
The embedding-projection pair for Var is similar: 

instance Generici Var where 
type Rep! Var = Repy ar 

from! (Var x) = Mi (U (Mi (Pa^ x))) 

from! (VarL xs) = Mi (R^ (M-^ (Comp! (gmap Red xs)))) 

td (Mi (Li (Mi (Pan x)))) = Var x 

toi (Mi (R, (Mt (Comp! xs)))) = VarL (gmap unRed xs) 

Note that composition is used both in the representation for the first argument of con- 
structor Let (of type [Decl p\) and in the nested recursion of VarL (of type Var [p]). 
In both cases, we have a recursive occurrence of a parameterised datatype where the 
parameter is not just the variable p. Recall our definition of composition: 
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data (:o:) cp ip p — Compi (cp (ip p)) 

The type <p is applied not to p, but to the result of applying ip to p. This is why we 
use :o: when the recursive argument to a datatype is not p, like in [ Decl p] and Var[p]. 
When it is p, we can simply use Req. 

We have seen how to represent many features of Haskell datatypes in our approach. 
We give a detailed discussion of the supported datatypes in |Section 1 1 .8.1 

1 1 .3 Generic functions 

In this section we show how to define type classes with a default generic implementation. 

11.3.1 Generic function definition 

Recall function encode from ISection 1 1 .21 

data Bit = O | I 

class Encode a where 
encode :: a — > [Bit] 

We cannot provide instances of Encode for our representation types, as those have kind 
* — > *, and Encode expects a parameter of kind *. We therefore define a helper class, 
this time parameterised over a variable of kind * — » *: 

class Encodei <p where 
encode! :: cp x [Bit] 

For constructors without arguments we return the empty list, as there is nothing to 
encode. Meta-information is discarded: 

instance Encode! Ui where 
encode! _ = [ 

instance (Encode! cp) => Encodei (Mi i y <p) where 
encode! (Mi a) = encode! a 

For a value of a sum type we produce a single bit to record the choice. For products we 
concatenate the encodings of both elements: 

instance (Encodei cp, Encodei 4>) Encodei (</> <A) where 
encodei (Li a) = O : encode! a 
encodei (Ri a) = I : encode! a 

instance (Encodei 0, Encodei 4>) Encodei (0 :x: (//) where 
encodei (a :x: b) = encode! a -H- encodei b 
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It remains to encode constants. Since constant types have kind *, we resort to Encode: 

instance (Encode </>) => Encode! (Ki i cf>) where 
encode! (Ki a) = encode a 

Note that while the instances for the representation types are given for the Encode! class, 
only the Encode class is exported and available for instantiation. This is because its type 
is more general, and because we need a two-level approach to deal with recursion: for 
the Ki instance, we recursively call encode instead of encodei . Recall our representation 
for Exp (simplified and with type synonyms expanded): 

type Repo Xp = Ki R Int :+: Ki R Exp :x: Ki R Exp 

Since Int and Exp appear as arguments to K 1; and our instance of Encode! for Ki ( 0 
reguires an instance of Encode </>, we need instances of Encode for Int and for Exp. We 



deal with Int in the next section, and Exp in Section 11.3.3 Finally, note that we do 



not need Encodei instances for Reci, Par! or :o:. These are only reguired for generic 



functions which make use of the Generic! class. We will see an example in Section 1 1 .3.4 



1 1 .3.2 Base types 

We have to provide the instances of Encode for the base types: 



instance Encode Int where encode = 
instance Encode Char where encode = 



Since Encode is exported, a user can also provide additional base type instances, or ad- 
hoc instances (types for which the reguired implementation is different from the derived 
generic behavior). 



11.3.3 Default definition 

We miss an instance of Encode for Exp. Instances of generic functions for representable 
types rely on the embedding-projection pair to convert from/to the type representation 
and then apply the generic function: 

encode De fauit " (Generic a, Encodei (Rep a)) => a -> [Bit] 
encode De fauit = encode! o from 

The function encodeDefauit i-S what we want to use as implementation of encode for types 
with a Generic instance. We want it to be the default implementation of encode: if the 
user does not provide an adhoc implementation, we should use this one. Unfortunately 
we cannot make it a standard Haskell 98 default, as in the following code: 

class Encode a where 
encode :: a — ► [Bit] 
encode = encodeDefauit 
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This code does not type-check, as the constraint of encodeoefauit ls not satisfied. Adding 
the constraint (Generic a, Encodei (Rep a)) to the Encode class is not a good solution, 
because it would prevent giving adhoc instances for types that are not instances of 
Generic. 

What we want is to say that the default method of encode should have a different type 
signature. This signature applies only to the default method, so it is only used when 
filling-in an instance that was left empty by the user. We call this a default signature: 

class Encode a where 
encode :: a — > [Bit] 

default encode :: (Generic a, Encode! (Rep a)) => a — > [ Bit] 
encode = encode! o from 

In this way we can give the desired default implementation for encode, together with 
the correct type signature. We have implemented default signatures as an extension to 
GHC; it is enabled with the pragma -XDef aultSignatures. 

Instantiating user datatypes is now a simple matter of giving an instance head. For 
instance, the instance of Encode for Exp is as follows: 

instance Encode Exp 

The instance for lists is similar, only that we need to specify the Encode constraint on 
the list argument: 

instance (Encode a) => Encode [ a] 



Default definitions are handled differently in UHC. We explain these in Section 11.5 



1 1 .3.4 Generic map 

In this section we define the generic map function gmap, which implements the Prelude's 
fmap function. Function gmap reguires access to the parameter in the representation 
type. As before, we export a single class, and use an internal class to define the generic 
instances: 

class G Functor <p where 

gmap :: (p —* a) — > c/> p — > (p a 

class G Functor! (p where 

gmap! :: (p —* a) — > <p p — > 4> a 

Unlike in Encode, the type arguments to GFunctor and GFunctori have the same kind, so 
we do not really need two classes. However, we argue that these are really two different 
classes. GFunctor is a user-facing class, available for being instantiated at any type cf> 
of kind * — > GFunctor!, on the other hand, is only used by the generic programmer 
for giving instances for the representation types. 
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We apply the argument function in the parameter case: 

instance GFunctor Par! where 
gmap! f (Par! a) = Pa^ (f a) 

Unit and constant values do not change, as there is nothing we can map over. We apply 
gmap! recursively to meta-information, sums, and products: 

instance GFunctor Ui where 
gmap! f Ui = 

instance GFunctor (K^ i y) where 

gmap! f (K^ a) = K, a 

instance (GFuncton 0) => GFunctor, (Mt i y 0) where 
gmap! f (M-t a) = M-t (gmap^ f a) 

instance (GFuncton 0, GFunctor! 0) => GFunctor (0 :+: 0) where 
gmap! f (Li a) = (gmap^ f a) 
gmap! f (Ri a) = Ri (gn^ f a) 

instance (GFunctor, 0, GFunctor 0) => GFuncton (0 :x: 0) where 
gmap 1 f (a :x: b) = gmap! f a :x: gmap 1 f b 

If we find a recursive occurrence of a functorial type we call gmap again, to tie the 
recursive knot: 

instance (GFunctor 0) => GFunctor (Rec! 0) where 
gmap! f (Rec-i a) = Rec^ (gmap f a) 

The remaining case is composition: 

instance (GFunctor 0, GFunctor 0) => GFunctor (0 :o: 0) where 
gmap! f (Comp! x) = Comp! (gmap (gmap! f) x) 

Recall that we reguire the first argument of :o: to be a user-defined datatype, and the 
second to be a representation type. Therefore, we use gmap! for the inner mapping (as 
it will map over a representation type) but gmap for the outer mapping (as it will reguire 
an embedding-projection pair). This is the general structure of the instance of :o: for a 
generic function. 

Finally we define the default method. It is part of the GFunctor class, which we have 
seen before, but we show the default only now because it depends on gmap^ 

default gmap :: (Generic 0, GFunctor (Rep! 0)) => (p -* a) -* 0 p -* cf> a 
gmap f = to-i o gmap^ f o from! 

Now GFunctor can be derived for user-defined datatypes. The usual restrictions apply: 
only types with at least one type parameter and whose last type argument is of kind * 
can derive GFunctor. Defining a GFunctor instance for lists is now very simple: 

instance GFunctor [] 

The instance GFunctor [] also guarantees that we can use [] as the first argument to :o:, 
as the embedding-projection pairs for such compositions need to use gmap. 
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11.3.5 Generic empty 

We can also easily express generic producers (functions which produce data). We will 
illustrate this with function empty, which produces a single value of a given type: 

class Empty a where 
empty :: a 

This function is perhaps the simplest generic producer, as it consumes no data. It relies 
only on the structure of the datatype to produce values. Other examples of generic 
producers are the methods in Read, the Arbitrary class from QuickCheck, and binary's 
get. As usual, we define an auxiliary type class: 

class Empty! cf> where 
empty! :: </> x 

Most instances of Empty! are straightforward: 

instance Empty! Ui where 
empty! = Ui 

instance (Emptyi 0) => Empty! (Mi i y 0) where 
empty! = Mi empty! 

instance (Emptyi 0, Emptyi 4>) Emptyi (</> :x: <P) where 
emptyi = emptyi :x: emptyi 

instance (Empty 4>) => Emptyi (Ki i </>) where 
emptyi = Ki empty 

For units we can only produce Ui. Meta-information is produced with Mi, and since we 
encode the meta-information using type classes (instead of using extra arguments to Mi) 
we do not have to use _L here. An empty product is the product of empty components, 
and for Ki we recursively call empty. The only interesting choice is for the sum type: 

instance (Emptyi </>) => Emptyi (0 '■+'■ <A) where 
emptyi = Li emptyi 

In a sum, we always take the leftmost constructor for the empty value. Since the left- 
most constructor might be recursive, function empty might not terminate. More complex 
implementations can look ahead to spot recursion, or choose alternative constructors 
after recursive calls, for instance. Note also the similarity between our Empty class and 
Haskell's Bounded: if we were defining minBound and maxBound generically, we could 
choose l_i for minBound and Ri for maxBound. This way we would preserve the seman- 



tics for derived Bounded instances, as defined by Peyton Jones 2003 , while at the same 
time lifting the restrictions on types that can derive Bounded. Alternatively, to keep the 
Haskell 98 behavior, we could give no instance for :x:, as enumeration types will not 
have a product in their representations. 

The default method (part of the Empty class) simply applies to to emptyi: 
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default empty :: (Generic a, Empty! (Rep a)) => a 
empty = to empty! 

Generic instances can now be defined: 

instance Empty Exp 

instance (Empty p) => Empty [p] 

Instances for other types are similar. 

11.3.6 Generic show 

To illustrate the use of constructor and selector labels, we define the shows function 
generically: 

class GShow a where 
gshows :: a — > ShowS 

gshow :: a —* String 
gshow x = gshows x " " 

We define a helper class GShow-i, with gshows! as the only method. For each represen- 
tation type there is an instance of GShow-|. The extra Bool argument will be explained 
later. Datatype meta-information and sums are ignored. For units we have nothing to 
show, and for constants we call gshows recursively: 

class GShow! <fi where 

gshowSi :: Bool — > 0 / — > ShowS 

instance (GShow! cj>) => GShow! (D^ y 0) where 

gshows! b (Mi a) = gshows! b a 
instance (GShow! (p, GShow! ip) => GShow! (cp :+: <//) where 

gshowSi b (U a) = gshows! b a 

gshows! b (Ri a) = gshows! b a 

instance GShow! Ui where 
gshowSi _ Ui = id 

instance (GShow <f>) => GShow! (K^ i </>) where 
gshows! _ (K-i a) = gshows a 

The most interesting instances are for the meta-information of a constructor and a selec- 
tor. For simplicity, we always place parentheses around a constructor and ignore infix 
operators. We do display a labeled constructor with record notation. At the constructor 
level, we use conlsRecord to decide if we print surrounding brackets or not. We use the 
Bool argument to gshowSi to encode that we are inside a labeled field, as we will need 
this for the product case: 
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instance (GShow! cp, Constructor y) => GShow! (d y <fi) where 
gshows 1 _c@ (Mi a) = showString "(" o showString (conName c) 
o showString " " 

o wrapR (gshows! (conlsRecord c) a o showString ")") 

where wrapR :: ShowS — » ShowS 

wrapR s | conlsRecord c = showString "{ " o s o showString " >" 
wrapR s | otherwise = s 

For a selector, we print Its label (as long as it Is not empty), followed by an equals sign 
and the value. In the product, we use the Bool to decide if we print a space (unlabeled 
constructors) or a comma: 

instance (GShow, cp, Selector y) => GShow! (S^ y (p) where 
gshows 1 b s@ (IVh a) 

| null (selName s) = gshows! b a 

| otherwise = showString (selName s) 

o showString " = " o gshows 1 b a 

instance (GShow! <p, GShow! ip) => GShow! (<p :x: (//) where 
gshows 1 b (a :x: c) = gshowSi b a 

o showString (if b then " , " else " ") 
o gshows 1 b c 

Finally, we show the default method: 

default gshows :: (Generic a, GShow! (Rep a)) => a -> ShowS 
gshows = gshowSi False o from 

We have shown how to use meta-information to define a generic show function. If 
we additionally account for infix constructors and operator precedence for avoiding un- 
necessary parentheses, we obtain a formal specification of how show behaves on most 
Haskell 98 datatypes. 



11.4 Compiler support 

We now describe in detail the required compiler support for our generic deriving mech- 
anism. 

We start by defining two predicates on types, isRepO </> and isRepI <p, which hold if cf> 
can be made an instance of Generic and Generici, respectively. The statement isRepO (f> 
holds if 0 is any of the following: 

1. A Haskell 98 datatype without context 

2. An empty datatype 

3. A type variable of kind * 
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We also reguire that for every type ip that appears as an argument to a constructor of 
<fi, isRepO ip holds. <fi cannot use existential guantification, type egualities, or any other 
extensions. 

The statement isRepI <p holds if the following conditions both hold: 

1. isRepO (p 

2. <p is of kind * — > * or k — > * — > *, for any kind k 



Note that isRepO holds for all the types of Section 11.2.4 while isRepI holds for [], Expr, 
Decl, and Var. 

Furthermore, we define the predicate ground <fi to determine whether a datatype has 
type variables. For instance, ground [Int] holds, but ground [a] does not. Finally, we 
assume the existence of an indexed fresh variable generator fresh pj, which binds p{ to a 
unigue fresh variable. 

For the remainder of this section, we consider a user-defined datatype 

data ... a„ = Co^ {I] :: p] I° 1 :: p° 1 } 

I Con m {X:: P : IgT " PS? 1 > 

with n type parameters, m constructors and possibly labeled parameter ij of type p{ at 
position j of constructor Con,. 



11.4.1 Type representation (kind *) 

|Figure 11.1| shows the generation of type representations for a datatype D satisfy- 
ing isRepO D. We generate a number of empty datatypes which we use in the meta- 
information: one for the datatype, one for each constructor, and one for each argument 
to a constructor that has a selector field. 

The type representation is a type synonym (Repg) with as many type variables as D. 
It is a wrapped sum of wrapped products: the wrapping encodes the meta-information. 
We wrap all arguments to constructors, even if the constructor is not a record. However, 
when the argument does not have a selector field we encode it with S NoSelector. Unlike 
the generated empty datatypes, NoSelector is exported to the user, who can match on it 
when defining generic functions. Since we use a balanced sum (resp. product) encoding, 
a generic function can use the meta-information to find out when the sum (resp. product) 
structure ends, which is when we reach (resp. S^. Each argument is tagged with 
Par 0 if it is one of the type variables, or Reco if it is anything else (type application or 
a concrete datatype). 



11.4.2 Generic instance 



Instantiation of the Generic class (introduced in Section 1 1 ,2| is defined in |Figure 1 1 .2 



The patterns of the from function are the constructors of the datatype applied to fresh 
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type Rep 0 D a, ... a n = D, $D (^ | m =1 (d SCon; (\~\* & (sel $Lj) (arg pj))))) 



data $D 

data SCon! 

data $Con m 
data $L] 

data $L° m 



sel $L{ 
arg pj 



Ij is defined 
otherwise 

n = 0 = 
n = 1 
otherwise 

n = 0 
n = 1 
otherwise 

3k e {1 ..n} 

otherwise 



$Lj 

NoSelector 



= V! 

= x 

= Ci x: + : Li=T x where m = Ln/2] 

= Ui 
= x 

= lTi x:x: nUTx where m = [n/2j 

: pj = a w = Par 0 pj 
= Rec 0 pj 



Figure 11.1: Code generation for the type representation (kind *) 



variables. The same patterns become expressions in function to. The patterns of to 
are also the same as the expressions of from, and they represent the different values 
of a balanced sum of balanced products, properly wrapped to account for the meta- 
information. Note that, for Generic, the functions tuple and wrap do not behave differently 
depending on whether we are in from or to, so for these declarations the dir argument is 
not needed. Similarly, the wrap function could have been inlined. These definitions will 
be refined in ISection 1 1 .4.4! 



11.4.3 Type representation (kind * — > *) 



|Figure 11 .3 shows the type representation of type constructors. We keep the sum-of- 



products structure and meta-information unchanged. At the arguments, however, we can 
use Par 0 , Par 1; Reco, Reci, or composition. We use Par! for the type variable a, and 
Par 0 for other type variables of kind *. A recursive occurrence of a type containing a n 
is marked with Reci. A recursive occurrence of a type with no type variables is marked 
with Reco, as there is no variable to abstract from. Finally, for a recursive occurrence 
of a type which contains anything other than a„ we use composition, and recursively 
analyse the contained type. 
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instance Generic (D a-\ ... a„) where 

type Rep (D a\ ... a„) — Rep° a-\ . . . a„ 

from pat'; 0 " 1 = exp'; 0 " 1 

pat f ™ m = exp f ™ m 
pat^ = exp 1 ? 

to pat" = exp'° 

exp?° = pat| rom = Con; (fresh p/) ... (fresh p?) 
exp| rom = patf = Mi (inj im (M 1 (tuple r P; 1 ... p° '))) 

inj, m x | m = 0 = _L 
| m = 1 = x 
| i < m' = Li (injj m . x) 

| i > m' = Ri (inj- ■ x) 

where m' = [m/2\ 

r = Li/2J 

tuplef ir p| . . . p°' | Oi = 0 
I o, = j 
| otherwise 
where k = [(o, /2)J 

wrap dir p = p 



from 
to 



= M-i U! 

= Mi (wrap dir (fresh pj)) 
= (tuple* p 1 ... pf) :x: (tuple* p|^ +1 ... p™) 



Figure 11.2: Code generation for the Generic instance 



11.4.4 Generic 1 instance 

The definition of the embedding-projection pair for kind * — > * datatypes, shown in 
|Figure 11.4 reflects the more complicated type representation. The patterns are un- 



changed. However, the expressions in fc^ need some additional unwrapping. This is 
encoded in var and unwC: an application to a type variable other than a n has been 
encoded as a composition, so we need to unwrap the elements of the contained type. 
We use gmap for this purpose: since we reguire isRepI <fi, we know that we can use 
gmap (see Section 1 1 .3.4) . The user should always derive GFunctor for container types, 



as these can appear to the left of a composition. 

Unwrapping is dual to wrapping: we use Par! for the type parameter a n , Reci for 
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type Rep° a, ... a n _, = D, $D (CL , (d SCon; (fl-L , (S, (sel $L|) (arg pj))))) 



Figure 11.3: Code generation for the type representation (kind * —> *) 

containers of cr,,, l^ for other type parameters and ground types, and composition for 
application to types other than a n . In ten we generate only Compi applied to a fresh 
variable, as this is a pattern; the necessary unwrapping of the contained elements is 
performed in the right-hand side expression. In from! the contained elements are tagged 
properly: this is performed by wC 0 . 

11.4.5 Meta-information 

We generate three meta-information instances. For datatypes, we generate: 

instance Datatype $D where 
moduleName _ = mName 
datatypeName _ = dName 

Here, dName is a String with the ungualified name of datatype D, and mName is a String 
with the name of the module in which D is defined. 
For constructors, we generate: 

instance Constructor SCon, where 
conName _ = name 
{conFixity _ = fixity} 
{conlsRecord _ = True} 

Here, t S {1..m}, and name is the ungualified name of constructor Con,. The braces 
around conFixity indicate that this method is only defined if Conj is an infix constructor. 
In that case, fixity is Infix assoc prio, where prio is an integer denoting the priority of Con,, 
and assoc is one of LeftAssociative, RightAssociative, or NotAssociative. These are derived 
from the declaration of Con, as an infix constructor. The braces around conlsRecord 
indicate that this method is only defined if Conj uses record notation. 



sel $Lj, 



YLm x ' anc ' njli x as in Figure 11.1 



arg pj 




pj = (j) a n A isRepI f) = Re^ pj 

pj = <p f2 A isRepI <f> A -i ground /? = (/> :o: arg /? 
otherwise = Rec 0 pj 
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instance Generic^ (D «i ... a„_i) where 

type Rep! (D a-\ ... a„-i) = Rep° a-\ 



a n -\ 



from! pat" 



from 



exp" L 



from! pat f ™ m 
td pat} 0 



exp, 
exp t0 



from 



td pat* = exp t0 
pat dir , expj rom , injj m x, and tuplef r p, . . . p m as 
exp[° = Coni(varpi) ... (var p 0i ) 



Figure 1 1 .2 (but using the new wrap dir x 



var pj 



wrap dir pj 



unwC,- 



wC Q 



Pj = 0 a 


A 


a ^ a„ 


A isRepI <j) — gmap unwC Q (fresh pj) 
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Figure 11.4: Code generation for the Generici instance 
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For all I G {1..m}, if a selector label ij is defined, we generate: 

instance Selector $L| where 
selName _ = l| 

Here, / G {1..o,}. If the selector label is not defined then the representation type uses 
NoSelector, which also has a Selector instance: 

data NoSelector 

instance Selector NoSelector where 
selName _ = "" 



11.5 Implementation In UHC 

So far we have presented the implementation of our technique in GHC. An earlier version 
of our work was implemented in UHC. In this section we detail the differences between 
the two implementations. 

The main reason behind the differences is UHC's lack of support for advanced type- 
level programming features, such as type families or functional dependencies. This 
makes the task of encoding the isomorphism between a datatype and its representa- 
tion more cumbersome. On the other hand, the UHC implementation reveals that a 
generic programming framework can still be integrated into a compiler even without 
many extensions beyond Haskell 98; the only significant extension we need is support 
for multi-parameter type classes. 

11.5.1 Datatype representation 

In UHC, we use a multi-parameter type class for embedding-projection pairs: 

class Representable 0 a t where 
from 0 :: a — > t x 
to 0 :: t x ^ a 

class Representable! 4> t where 
from! :: <f> p — > t p 
toi :: t p — > 0 p 

The variable t encodes the representation type. It is uniquely determined by a (and </>), 
but we avoid the use of functional dependencies. Note also the different naming of the 
classes and their methods. 

Instantiation remains mostly unchanged, apart from the positioning of the represen- 
tation type. See, for example, the instantiation of lists (using the types defined in 
Section 1 1 .274) : 
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instance Representable 0 [p] (Repg st p) where 

fromoO = Mi (L, {M, Ui)) 

from 0 (h : t) = Mi (Mi (K, h :x: t))) 

too (Mi (U (Mi Ui))) = [] 

too (M, (R, (Mi (Ki h :x: t)))) = h : t 

instance Representable! [] Rep^' where 

from! [] = Mi (L, (Mi U^) 

from! (h : t) = (R^ (M, (Pan h :x: Rec! t))) 

toi (Mi (U (Mi U^)) = 0 

toi (Mi (Ri (Mi (Pan h :x: ReCi t)))) = h : t 

In particular, the code for the methods is exactly the same. 
11.5.2 Specifying default methods 

UHC does not have support for default signatures, so we cannot use those. Instead, we 
define the default definition of each generic function separately, and then tie it together 
with its class using a pragma. For instance, for Encode the generic function developer 
has to define: 

encode Defau | t :: (Representable 0 a t, Encode! t) => t x — > a — > [ Bit] 
encode De fauit rep x = encode! ((from 0 x) 'asTypeOf rep) 

Because we do not use functional dependencies, we have to pass the representation 
type explicitly to the function encodeoefauit- This function then uses the representation 
type to coerce the result type of from with asTypeOf. 
The instances of Encode for Exp and [] are as follows: 

instance Encode Exp where 

encode = encode Defau i, (1 :: Repo"" x) 

instance (Encode p) => Encode [ p ] where 
encode = encode Defau i, (1 :: Rep^ ist p x) 

Note, however, that the instance for [] reguires scoped type variables to type-check. We 
can avoid the need for scoped type variables if we create an auxiliary local function 
encodej] with the same type and behavior of encodeoefauit : 

instance (Encode p) => Encode [p] where 
encode = encode^ _L where 

encode]] :: (Encode p) => Repo' st p / — > [p] — > [Bit] 
encode^ = encode Defau | t 

Here, the local function encode^ encodes in its type the correspondence between the 
type [p] and its representation RepQ lst p. Its type signature is reguired, but can easily 



156 



11.5 Implementation in UHC 



be obtained from the type of encodeoefauit by replacing the type variables a and t with 
the concrete types for this instance. 

These instances are considerably more complicated than the empty instances we 
use in GHC, but they are automatically generated by UHC. However, we still need a 
mechanism to tie the default definitions to their respective classes. We use a pragma 
for this purpose: 

{— # DERIVABLE Encode encode encode De fauit #— } 
This pragma takes three arguments, which represent (respectively): 

1. The class which we are defining as derivable 

2. The method of the class which is generic (and therefore needs a default definition) 

3. The name of the function which serves as a default definition 

Since a class can have multiple generic methods, multiple pragmas can be used for this 
purpose. 

11.5.3 Instantiating generic functions 

After the generic programmer has defined encodeoefauit ar| d given the pragma, the user 
still has to specify that a specific datatype should get a generic instance. Unlike in 
GHC, in UHC we do use deriving for this purpose. So, given a generic function f that is 
a method of the type class F, and for every datatype D that derives F with type arguments 
CT| ... a n and associated representation type Rep° a-\ ... a n x, the compiler generates: 

instance Ctx => F (D a-\ ... a n ) where 

f = f D 1 

where f D :: Ctx => Repo a-\ . . . a n x —> P 

fp — fDefault 

The type ft is the type of f specialised to D, and / is a fresh type variable. The context 
Ctx is the same in the instance head and in function f D . The exact context generated 
depends on the way the user specifies the deriving. If deriving F is attached to the 
datatype, we generate a context F a-\ , . . . , F a„, where ~a is the variable a applied to 
enough fresh type variables to achieve full saturation. This approach gives the correct 
behavior for Haskell 98 derivable classes like Show. 

In general, however, this is not correct: we cannot assume that we reguire F a, for all 
( £ {1..n}: some generic functions do not reguire any constraints because they do not 
recurse into subterms. Even worse, we might reguire constraints other than these, as a 
generic function can use other functions, for instance. 

To avoid these problems we can use the standalone deriving extension, which is 
supported by UHC. The important difference between standalone deriving and standard 
deriving is that the user supplies the context directly: 
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deriving instance (Ctx) => F (D a-\ ... a n ) 

Then we can simply use this context for the instance. 

The use of the deriving clause left us in a unsatisfactory situation. While simply 
attaching deriving F to a datatype declaration is a simple and concise syntax, it does 
not always work, because in some cases the compiler cannot infer the right context, and 
as such the user is forced to supply a standalone deriving instance. It does, however, 
follow the behavior of deriving for the Haskell 98 derivable classes, as both UHC and 
GHC cannot always infer the right context (for instance for nested datatypes). In contrast, 
our use of default method signatures and empty instance declarations in GHC leads to 
a more consistent instantiation strategy. 



1 1 .6 Alternatives 

We have described how to implement a deriving mechanism that can be used to specify 
many datatype-generic functions in Haskell. There are other alternatives, of varying 
complexity and type-safety. 



11.6.1 Pre-processors 

The simplest, most powerful, but least type safe alternative to our approach is to im- 
plement deriving by pre-processing the source file(s), analysing the datatypes defini- 



tions and generating the reguired instances with a tool such as DrIFT |Winstanley and 



Meacham 2008. This reguires no work from the compiler writer, but does not simplify 
the task of adding new derivable classes, as programming by generating strings is not 
very convenient. 

Staged meta-programming {Section 2.1 lies in between a pre-processor and an em- 



bedded datatype-generic representation. GHC supports Template Haskell |Sheard and 



Peyton Jones 2002 , which has become a standard tool for obtaining reflection in Haskell. 
While Template Haskell provides possibly more flexibility than the purely library-based 
approach we describe, it imposes a significant hurdle on the compiler writer, who has 
not only to implement a language for staged programming (if one does not yet exist for 
the compiler, like in UHC), but also to keep this complex component up-to-date with 
the rest of the compiler, as it evolves. As evidence of this, Template Haskell support for 
GADTs and type families only arrived much later than the features themselves. Also, for 
the derivable class writer, using Template Haskell is more cumbersome and error-prone 
than writing a datatype-generic definition in Haskell itself. 

For these reasons we think that our library-based approach, while having some limi- 
tations, offers a good balance between expressive power, type safety, and the amount of 
implementation effort reguired. 
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11.6.2 Library choice 

Another design choice we made was In the specific library approach to use. We 
have decided not to use any of the existing libraries but Instead to develop yet an- 
other one. However, our library Is merely a variant of existing libraries, from which 
It borrows many Ideas. We see our representation as a mixture between polyp and 
instant-generics. We share the functorlal view with polyp, but we abstract from 
a single type parameter, and not from the recursive occurrence. Our library can also 
be seen as instant-generics extended with a single type parameter. Having one 
parameter allows us to deal with composition effectively, and we do not duplicate the 
representation for types without parameters. 



1 1 .7 Related work 



Clean Allmarlne and Plasmeljer 2001 has also Integrated generic programming directly 



In the language. We think our approach Is more lightweight: we express our generic 
functions almost entirely In Haskell and regulre only one small syntactic extension. 
On the other hand, the approach taken In Clean allows defining generic functions with 



polyklnded types |Hlnze 2002 , which means that the function bimap (see Section 1 1 .2.1 



for Instance, can be defined. Not all Clean datatypes are supported: guantlfled types, 
for example, cannot derive generic functions. Our approach does not support all features 
of Haskell datatypes, but most common datatypes and generic functions are supported. 

An extension for derivable type classes similar to ours has been developed by |Hlnze" 
and Peyton Jones 2001 In GHC As In Clean, this extension regulres special syntax 
for defining generic functions, which makes It harder to Implement and maintain. In 
contrast, generic functions written In our approach are portable across different compil- 



ers. Furthermore, Hlnze and Peyton Jones s approach cannot express functions such as 
gmap, as their type representation does not abstract over type variables. Since GHC 
version 7.2.1, support for derivable type classes has been dropped, and replaced by the 
extension we describe In this chapter. 



Rodriguez Yakushev et a I. 2008 give criteria for comparing generic programming li- 



braries. These criteria consider the library's use of types, and Its expressiveness and 
usability. Regarding types, our library scores very well: we can represent regular, 
hlgher-klnded, nested, and mutually recursive datatypes. We can also express subunl- 
verses: generic functions are only applicable to types that derive the corresponding 
class. We only lack the ability to represent nested hlgher-klnded datatypes, as our 
representation abstracts only over a parameter of kind *. 

Regarding expressiveness, our library scores good for most criteria: we can abstract 
over type constructors, give ad-hoc definitions for datatypes, our approach Is extensible, 
supports multiple generic arguments, represents the constructor names and can express 
consumers, transformers, and producers. We cannot express gmapQ In our approach, but 
our generic functions are still first-class: we can call generic map with generic show as 
argument, for Instance. Ad-hoc definitions for constructors would be of the form: 
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instance Encode Exp where 

encode (Plus ei 62) = ... — code specific to this constructor 
encode x = encodeoefauit 

Regarding usability, our approach supports separate compilation, is highly portable, has 
automatic generation of its two representations, reguires minimal work to instantiate and 
define a generic function, is implemented in a compiler and is easy to use. We have 
not benchmarked our library in UHC. As for GHC, we discuss performance matters of a 



similar approach in detail in Chapter 9 



1 1 .8 Future work 

Our solution is applicable to a wide range of datatypes and can express many generic 
functions. However, some limitations still remain, and many improvements are possible. 
In this section we outline some possible directions for future research. 



11.8.1 Supported datatypes 

Our examples in |Section 11. 2| show that we can represent many common forms of 
datatypes. We believe that we can represent all of the Haskell 98 standard datatypes 
in Generic, except for constrained datatypes. We could easily support constrained 
datatypes by propagating the constraints to the generic instances. 

Regarding Generici, we can represent many, but not all datatypes. Consider a nested 
datatype for representing balanced trees: 

data Perfect p = Node p | Perfect (Perfect (p , p)) 

We cannot give a representation of kind * —> * for Perfect, since for the Perfect construc- 
tor we would need something like Perfect :o: Reci ((,) p). However, the type variable p 
is no longer available, because we abstract from it. This limitation is caused by the fact 



that we abstract over a single type parameter. The approach taken by Hesselink 2009 
is more general and fits closely with our approach, but it is not clear if it is feasible 
without advanced language extensions. 

Note that for this particular case we could use a datatype which pairs elements of a 
single type: 

data Pair p = Pair p p 

The representation for the Perfect constructor could then be Perfect :o: Reci Pair. 



11.8.2 Generic functions 

The representation types we propose limit the kind of generic functions we can define. 
We can express the Haskell 98 standard derivable classes Eq, Ord, Enum, Bounded, 
Show, and Read, even lifting some of the restrictions imposed on the Enum and Bounded 
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instances. All of these are expressible for Generic types. Using Generic^ we can imple- 
ment Functor, as the parameter of the Functor class is of kind *—»•*. The same holds 
for Foldable and Traversable. For Typeable we can express Typeable 0 and Typeablei. 

On the other hand, the Data class has very complex generic functions which cannot be 
expressed with our representation. Function gfoldl, for instance, reguires access to the 
original datatype constructor, something we cannot do with the current representation. 
We plan to explore if and how we can change our representation to allow us to express 
more generic functions, also in the light of the conclusions of our formal comparison of 
Chapter 8 



1 1 .9 Conclusion 

We have shown how generic programming can be better integrated in Haskell by revis- 
iting the deriving mechanism. All Haskell 98 derivable type classes can be expressed 
as generic functions in our library, with the advantage of becoming easily readable and 
portable. Additionally, many other type classes, such as Functor and Foldable, can be de- 
fined generically. Adding new classes with generic methods can now be done by generic 
programmers in regular Haskell; previously, this would be the compiler developer's task, 
and would be done using code generation, which is more error-prone and verbose. 

We have implemented our solution in both UHC and GHC and invite users to ex- 
periment with this feature. We hope our work paves the way for a redefinition of the 
behavior of derived instances for Haskell Prime |Wallace et a I. 2007 . 
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Epilogue 



This thesis is about abstraction. More concretely, it is about one form of abstraction in 
programming languages: generic programming. Abstraction is "a good thing": it allows 
the programmer to focus on the essential parts of the problem, and promotes code reuse 
by making fewer assumptions on underlying data structure. 

Yet, abstraction is hard. It is easier to explain to a child why 2 + 2 is 4 than to explain 
the addition of any two arbitrary numbers. It is easier for a programmer to define a 
function that works on a single datatype than to define a function that works on a large 
class of datatypes. 

However hard generic programming might be, its benefits are enormous. Being forced 
to define functionality for each datatype, and to adapt this functionality whenever a 
datatype is changed, leads programmers into avoiding the use of many datatypes. A 
single, universal type, that can accommodate many different types of structure, each with 
different invariants, might be preferred to having to deal with a number of more specific 
datatypes. This can guickly lead to a programming style where many advantages of 
a static type system are compromised, since types stop being informative and diverse. 
On the other hand, being free from adapting functionality to each datatype leads to 
explorative use of the type system; using many different datatypes is not a burden, since 
much functionality comes "for free". We have seen in a non-trivial practical application of 
generic programming that this freedom enables faster and more productive development, 



especially during initial development and prototyping Magalhaes and De Haas 2011 



We want to help preventing programmers from reducing type diversity in their code, as 
we believe this cripples program development. Unfortunately, many programmers are not 
even aware that conflating different types, each with detailed structure and invariants, 
under a single "umbrella type", with loose structure and semantics, is a bad thing. After 
all, it is a way of increasing abstraction; however, it also increases the chances for things 
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to go wrong, and for subtle bugs caused by unsatisfied data Invariants to creep In. The 
only solution for this Is to Increase programmers' awareness of the alternative: generic 
programming. Driving adoption of a programming paradigm regulres a lot of distributed 
effort, and Involves tasks such as language design, compiler development or adaptation, 
case studies, development of large and realistic examples, and Inclusion In educational 
curricula. 

We have contributed to all of these tasks, In one way or another. We started by Intro- 
ducing generic programming as an abstraction mechanism, and then toured through sev- 
eral different approaches to generic programming, gaining knowledge about the power, 
usefulness, and limitations of each approach by modelling and formally relating the 
approaches. This Is an essential aspect of understanding generic programming, and 
answers the first research guestlon. With this knowledge, we took off to the practical 
realm, and addressed limitations that often prevent adoption of generic programming: 
added more datatype support, Improved runtime efficiency, and showed how to integrate 
it natively In the language for better usability. These are all ways to Improve the prac- 
tical application of generic programming, and address the second research guestlon. 



We have also developed case studies and applications of generic programming Jeurlng 



et al. 2010 Magalhaes and De Haas 2011 , but have not Included them In this thesis 



At the same time, we have taken part In the educational efforts at Utrecht University, 
teaching students about generic programming at the graduate level. These efforts prove 
fruitful, as previous students have been using generic programming successfully In a 
number of startup companies, such as Well-Typed |htt p : / / well- typed . com/) , Vector 
Fabrics (http : //v ectorf abric s . com/} , and Silk |http : / / silkap p . com/) . 

We believe It Is safe to say that Haskell, and conseguently wider adoption of generic 
programming, Is at a turning point. With multlcore processors becoming commonplace 
In computers, Interest In parallel computing has surged. Traditional Imperative program- 
ming languages provide very low-level support for parallelism, while Haskell has great 



potential for providing simple and mostly automated parallellsation Peyton Jones and 



Singh 


2009 


crucia 


ly on 



In particular, automated data parallelism Peyton Jones et al. 2008 relies 



cally. The growing evidence of Haskell as a practical programming language for parallel 
applications, together with the maturity of Its libraries and tools through the Haskell 
Platform effort |http://hackage. haskell.org/platform/ 1, will bring the language 
to the attention of a much wider audience. 

Large scale adoption of generic programming will bring a number of new challenges 
to the field. So far, most approaches have been experimental and research-oriented, 
poorly documented, and rarely ever supported for a long time. A generic programming 
approach suited for Industrial adoption will need to be not only fast and easy to use, but 
also reliable, well documented, and maintained, keeping a stable Interface. Our work of 
Integrating a generic programming library In the compiler {Chapter 11) provides a good 
basis for an "Industrial-strength" approach: It Is easily available because It comes with 
the compiler, It Is fast, easy to use, and can be updated while keeping a stable Interface 
because It Is supported directly In the language. 

The Increased popularity of Haskell does not mean that future efforts In generic 
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programming will be limited to engineering issues. The advent of practical dependently- 
typed programming languages brings new opportunities for generic programming, such 
as representation of value-dependent datatypes, instantiation of generic functions to 
these datatypes, and generic proofs, to name a few. Dependently-typed programming 
certainly allows for increased abstraction, and the higher complexity of the type system 
means that more information is available at the type level; this translates to increased 
opportunities for generic programming by exploiting the structure of types. 

We look forward to the exciting challenges the future will bring to generic program- 
ming as a programming practice. Generic programming is not only desirable but also 
necessary for the development of high guality statically-typed functional programs, and 
increasingly complex software development reguires increasingly abstract programming 
technigues. As such, we remain committed to facilitating the use of generic programming 
and evaluating its success in improving the guality of programs, for a future where less 
is more: less code, but more reliable, simple, and correct code. 
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